@ckanner/flywheel 0.31.0 → 0.31.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +48 -5
- package/CONTRIBUTING.md +19 -0
- package/LICENSE +21 -0
- package/README.en.md +4 -171
- package/README.md +152 -269
- package/README.zh-CN.md +116 -0
- package/bin/flywheel-claude-code-pro-worker.js +1 -1
- package/bin/flywheel-codex-pro-worker.js +1 -1
- package/bin/flywheel-dsh-worker.js +1 -1
- package/bin/flywheel.js +1 -1
- package/bin/runtime.js +1 -1
- package/docs/README.md +84 -0
- package/docs/README.zh-CN.md +64 -0
- package/docs/archive/README.md +13 -0
- package/docs/archive/codex-pro-adapter-handoff.md +197 -0
- package/docs/archive/design-v0.24.md +540 -0
- package/docs/design.md +681 -0
- package/docs/en/claude-code-pro.md +25 -16
- package/docs/en/configuration.md +306 -141
- package/docs/en/contributing.md +303 -0
- package/docs/en/flow-console.md +67 -13
- package/docs/en/getting-started.md +232 -126
- package/docs/en/investigation.md +219 -0
- package/docs/en/jev-shadow.md +67 -0
- package/docs/en/model-policy.md +147 -62
- package/docs/en/releasing.md +207 -0
- package/docs/en/session-evaluation.md +4 -2
- package/docs/en/session-recovery.md +49 -6
- package/docs/en/workflow.md +222 -114
- package/docs/evaluation/README.md +12 -0
- package/docs/evaluation/jev-historical-provisional.json +138 -0
- package/docs/evaluation/jev-historical-result-2026-11-10.json +174 -0
- package/docs/plans/2026-11-01-dsh-integration.md +67 -0
- package/docs/plans/2026-11-01-model-profiles-preflight.md +29 -0
- package/docs/plans/2026-11-01-npm-installation.md +19 -0
- package/docs/plans/2026-11-03-codex-pro-default.md +28 -0
- package/docs/plans/2026-11-06-adaptive-workflow.md +365 -0
- package/docs/plans/2026-11-07-session-lifecycle.en.md +424 -0
- package/docs/plans/2026-11-07-session-lifecycle.md +337 -0
- package/docs/plans/2026-11-08-flow-console.md +271 -0
- package/docs/plans/2026-11-10-jev-shadow.md +32 -0
- package/docs/plans/2026-11-10-task-discovery.md +17 -0
- package/docs/plans/2026-11-10-workbench-redesign.md +19 -0
- package/docs/plans/2026-11-12-feedback-driven-workflows.md +739 -0
- package/docs/plans/README.md +22 -0
- package/docs/reviews/2026-08-24-v0.18-review-fixplan.md +265 -0
- package/docs/reviews/2026-09-12-pr4-pr5-review.md +70 -0
- package/docs/reviews/README.md +12 -0
- package/docs/validation/2026-11-01-npm-installation.md +105 -0
- package/docs/validation/2026-11-01-profiles-preflight.md +67 -0
- package/docs/validation/2026-11-01-results.md +114 -0
- package/docs/validation/2026-11-03-codex-pro-default.md +90 -0
- package/docs/validation/2026-11-04-dsh-native-view.md +85 -0
- package/docs/validation/2026-11-04-release-0.28.1.md +33 -0
- package/docs/validation/2026-11-04-release-0.28.2.md +35 -0
- package/docs/validation/2026-11-05-dsh-0.2.2-compatibility.md +26 -0
- package/docs/validation/2026-11-05-registry-smoke-recovery.md +26 -0
- package/docs/validation/2026-11-05-release-0.28.3.md +32 -0
- package/docs/validation/2026-11-05-subscription-model-access.md +60 -0
- package/docs/validation/2026-11-05-view-order.md +14 -0
- package/docs/validation/2026-11-07-adaptive-workflow.md +79 -0
- package/docs/validation/2026-11-07-claude-code-pro.md +144 -0
- package/docs/validation/2026-11-07-session-lifecycle.md +136 -0
- package/docs/validation/2026-11-08-session-p3-p4.md +99 -0
- package/docs/validation/2026-11-09-flow-console.md +81 -0
- package/docs/validation/2026-11-10-jev-shadow.md +50 -0
- package/docs/validation/2026-11-10-task-discovery.md +45 -0
- package/docs/validation/2026-11-10-workbench-redesign.md +54 -0
- package/docs/validation/2026-11-11-release-0.30.2.md +65 -0
- package/docs/validation/2026-11-11-release-0.31.0.md +46 -0
- package/docs/validation/2026-11-12-dsh-0.2.2-rc.2.md +58 -0
- package/docs/validation/2026-11-12-feedback-driven-workflows.md +187 -0
- package/docs/validation/README.md +41 -0
- package/docs/validation/evidence/2026-11-08-session-p3-p4.json +5430 -0
- package/docs/validation/evidence/claude-node-bridges.txt +118 -0
- package/docs/validation/evidence/claude-npm-build.txt +15 -0
- package/docs/validation/evidence/claude-npm-candidate.json +54 -0
- package/docs/validation/evidence/claude-npm-install.txt +21 -0
- package/docs/validation/evidence/claude-npm-tests.txt +145 -0
- package/docs/validation/evidence/claude-plugin-tests.txt +205 -0
- package/docs/validation/evidence/claude-quality-gate.txt +29 -0
- package/docs/validation/evidence/claude-race.txt +7 -0
- package/docs/validation/evidence/codex-pro-070-probes.txt +4 -0
- package/docs/validation/evidence/dsh-bridge.txt +4 -0
- package/docs/validation/evidence/dsh-native-view-bridge-tests.txt +1 -0
- package/docs/validation/evidence/dsh-native-view-build.txt +15 -0
- package/docs/validation/evidence/dsh-native-view-candidate.json +54 -0
- package/docs/validation/evidence/dsh-native-view-inspect.txt +27 -0
- package/docs/validation/evidence/dsh-native-view-install-tests.txt +19 -0
- package/docs/validation/evidence/dsh-native-view-npm-tests.txt +143 -0
- package/docs/validation/evidence/dsh-native-view-plugin-tests.txt +99 -0
- package/docs/validation/evidence/dsh-native-view-quality-gate.txt +29 -0
- package/docs/validation/evidence/dsh-plugin-tests.txt +48 -0
- package/docs/validation/evidence/function-coverage.txt +378 -0
- package/docs/validation/evidence/npm-build.txt +13 -0
- package/docs/validation/evidence/npm-contract-tests.txt +65 -0
- package/docs/validation/evidence/npm-dsh-bridge.txt +18 -0
- package/docs/validation/evidence/npm-dsh-plugin-tests.txt +62 -0
- package/docs/validation/evidence/npm-function-coverage.txt +420 -0
- package/docs/validation/evidence/npm-install-tests.txt +20 -0
- package/docs/validation/evidence/npm-quality-gate.txt +29 -0
- package/docs/validation/evidence/npm-race.txt +7 -0
- package/docs/validation/evidence/pr-dsh-bridge.txt +4 -0
- package/docs/validation/evidence/pr-dsh-plugin-tests.txt +48 -0
- package/docs/validation/evidence/pr-function-coverage.txt +379 -0
- package/docs/validation/evidence/pr-quality-gate.txt +28 -0
- package/docs/validation/evidence/pr-race.txt +6 -0
- package/docs/validation/evidence/profiles-control-smoke.json +1 -0
- package/docs/validation/evidence/profiles-dsh-bridge.txt +18 -0
- package/docs/validation/evidence/profiles-dsh-plugin-tests.txt +59 -0
- package/docs/validation/evidence/profiles-function-coverage.txt +411 -0
- package/docs/validation/evidence/profiles-quality-gate.txt +29 -0
- package/docs/validation/evidence/profiles-race.txt +7 -0
- package/docs/validation/evidence/quality-gate.txt +28 -0
- package/docs/validation/evidence/race.txt +6 -0
- package/docs/validation/evidence/watchdog-startup-after.txt +2 -0
- package/docs/validation/evidence/watchdog-startup-before.txt +6 -0
- package/docs/validation/evidence/watchdog-startup-quality-gate.txt +29 -0
- package/docs/validation/evidence/watchdog-startup-race.txt +7 -0
- package/docs/{claude-code-pro.md → zh-CN/claude-code-pro.md} +23 -17
- package/docs/{configuration.md → zh-CN/configuration.md} +323 -139
- package/docs/zh-CN/contributing.md +240 -0
- package/docs/{flow-console.md → zh-CN/flow-console.md} +55 -12
- package/docs/zh-CN/getting-started.md +352 -0
- package/docs/zh-CN/investigation.md +182 -0
- package/docs/zh-CN/jev-shadow.md +79 -0
- package/docs/zh-CN/model-policy.md +123 -0
- package/docs/zh-CN/releasing.md +120 -0
- package/docs/{session-evaluation.md → zh-CN/session-evaluation.md} +4 -2
- package/docs/{session-recovery.md → zh-CN/session-recovery.md} +46 -12
- package/docs/zh-CN/workflow.md +270 -0
- package/npm/install.js +1 -1
- package/package.json +12 -25
- package/docs/getting-started.md +0 -418
- package/docs/model-policy.md +0 -65
- package/docs/workflow.md +0 -314
|
@@ -0,0 +1,219 @@
|
|
|
1
|
+
# Bounded investigation and feedback-driven workflows
|
|
2
|
+
|
|
3
|
+
[Documentation](../README.md) · English · [简体中文](../zh-CN/investigation.md) · [Workflow](workflow.md)
|
|
4
|
+
|
|
5
|
+
Task is the recommended goal entry. It identifies the requested deliverable and
|
|
6
|
+
prepares the appropriate contract. Auto chooses Quick or Full only after a valid
|
|
7
|
+
implementation contract exists. An investigation can deliver a report on its own;
|
|
8
|
+
it is not a third implementation completion mode.
|
|
9
|
+
|
|
10
|
+
| Daily request | Entry | Engine behavior and endpoint |
|
|
11
|
+
| --- | --- | --- |
|
|
12
|
+
| Diagnose and explain | `/flywheel-task` | Managed reads/searches and an evidence report; no implementation |
|
|
13
|
+
| Diagnose and repair within scope | `/flywheel-task` | Supported report, then Spec and Auto; retain provenance, usage and scope |
|
|
14
|
+
| Repair an identified issue | Task Auto or explicit Quick | Qualification, mandatory checks, independent final Review and Git delivery |
|
|
15
|
+
| Propose a complex change for review | `/flywheel-new` | Stop after preparation; Run Full after implementation authorization |
|
|
16
|
+
| Continue an existing task | `/flywheel-run` | Resume its original preparation or implementation ledger |
|
|
17
|
+
| Run a particular stage | Plan / Exec / Verify / Review | Direct controls remain; stage success is not whole-task completion |
|
|
18
|
+
|
|
19
|
+
Kimi uses `/skill:flywheel-*`. Status and Stop work for investigations and
|
|
20
|
+
implementation. Existing repair authorization is retained; new scope or an
|
|
21
|
+
unresolved product decision needs a separate decision.
|
|
22
|
+
|
|
23
|
+
## Preparation contract
|
|
24
|
+
|
|
25
|
+
The host prepares an input from the user's goal and repository facts. Go validates
|
|
26
|
+
it and owns `preparation.json`. Users do not need to fill another large form, and
|
|
27
|
+
the host never edits preparation state or `state.json` directly. The installed CLI
|
|
28
|
+
exposes `preparation.schema.json`, `preparation-state.schema.json` and
|
|
29
|
+
`preparation.example.json` through `flywheel template`.
|
|
30
|
+
|
|
31
|
+
Version 1 records the goal, expected result, environment, time window, known
|
|
32
|
+
evidence, falsifiable hypotheses, exact read paths, concrete actions and budgets.
|
|
33
|
+
`intent` is `investigate` or `investigate-and-fix`. Repair intent requires a nonempty
|
|
34
|
+
`implementation-scope`; report-only intent grants no implementation scope. A Spec
|
|
35
|
+
is not required while only preparation exists.
|
|
36
|
+
|
|
37
|
+
The initial action family is implemented directly by Go:
|
|
38
|
+
|
|
39
|
+
| kind | Capability | Boundaries |
|
|
40
|
+
| --- | --- | --- |
|
|
41
|
+
| `read_file` | Read one approved UTF-8 file | Reject symlinks, special files and excessive input |
|
|
42
|
+
| `search_text` | Literal search in one approved file, with line numbers | No regular expressions or command interpolation |
|
|
43
|
+
| `list_files` | List names in one approved directory | No recursive content grant; at most 4096 names |
|
|
44
|
+
|
|
45
|
+
Paths must be clean repository-relative paths. Each action references an approved
|
|
46
|
+
path and existing hypotheses. The opened inode must match the inspected inode;
|
|
47
|
+
`os.Root` confines traversal. A directory grant permits names only. This version
|
|
48
|
+
has no arbitrary shell, external query, production restart, cache flush, data
|
|
49
|
+
repair or writable investigator Agent action. Existing full-access configuration
|
|
50
|
+
does not add capabilities to these Go handlers. Broader experiments require
|
|
51
|
+
additional handlers, contracts and validation.
|
|
52
|
+
|
|
53
|
+
`limits` freezes action count, overall deadline, per-file bytes and total output
|
|
54
|
+
bytes. Resume does not reset them. The deadline starts at preparation, including
|
|
55
|
+
paused time. Bounded UTF-8 excerpts retain a content digest; `truncated` means the
|
|
56
|
+
display is incomplete, not that an entire log or test inventory was observed.
|
|
57
|
+
|
|
58
|
+
## CLI and desktop controls
|
|
59
|
+
|
|
60
|
+
Create the actual task directory, then submit a complete preparation request:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
flywheel control investigate --task-dir "$task_dir" --cwd "$repo_dir" < start-request.json
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
The outer request is `{ "action": "start", "contract": <complete-contract> }`;
|
|
67
|
+
read the installed example and replace its paths, goals and permissions. `start`
|
|
68
|
+
saves ready preparation without executing actions. To run, submit
|
|
69
|
+
`{ "action": "run", "revision": <observed-revision> }` through the same command.
|
|
70
|
+
|
|
71
|
+
Go starts a background owner under the existing task lifetime lease and token-bound
|
|
72
|
+
stop protocol. In DSH, `flywheel_control` accepts `action: investigate` and the nested
|
|
73
|
+
`investigation` request, associating the exact task with the conversation. The panel
|
|
74
|
+
shows budgets, observations, excerpts, decisions and reports, with continue,
|
|
75
|
+
reconcile, stop and refresh controls.
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
flywheel control status --task-dir "$task_dir" --cwd "$repo_dir"
|
|
79
|
+
flywheel status "$task_dir"
|
|
80
|
+
flywheel stop "$task_dir"
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
`ok:true` acknowledges a control request. `running` comes from the lifetime lease.
|
|
84
|
+
An investigation-only response contains `preparation`, not a fabricated code-task
|
|
85
|
+
`state`. Reconcile an uncertain result in the original directory before mutation.
|
|
86
|
+
|
|
87
|
+
## Report and repair handoff
|
|
88
|
+
|
|
89
|
+
`awaiting-report` means observations are available. The current host model submits
|
|
90
|
+
one evidence-based report; Go checks fields, hypothesis coverage and references.
|
|
91
|
+
Separate facts from inference, classify each hypothesis as supported, contradicted
|
|
92
|
+
or unknown, and include counter-evidence, limitations, recommendation, regression
|
|
93
|
+
checks and rollback conditions. References are real observation `invocation.id`
|
|
94
|
+
values; missing or failed observations cannot be cited as successful evidence.
|
|
95
|
+
|
|
96
|
+
Submit `action: report`, the observed `revision` and `report` through
|
|
97
|
+
`control investigate`. Terminal report states are `supported`, `inconclusive` and
|
|
98
|
+
`access-blocked`; cancellation produces `cancelled`. Supported means the analyst's
|
|
99
|
+
conclusion has structurally valid evidence references. Go does not independently
|
|
100
|
+
prove semantic causality. Uncertainty produces a report with limitations, not an
|
|
101
|
+
automatic expansion of investigation or implementation authority.
|
|
102
|
+
|
|
103
|
+
When the original request authorized repair and the investigation is supported
|
|
104
|
+
with all receipts resolved, prepare Spec/packets in the same workspace and task
|
|
105
|
+
directory, then launch normally. `preparation-source` records provenance, the report
|
|
106
|
+
digest, authorized implementation scope, actions and elapsed time. Known JEV usage
|
|
107
|
+
is imported once on its original dispatch day; missing usage and cost remain
|
|
108
|
+
unknown. Resume never double-counts. Actual implementation changes are checked
|
|
109
|
+
against the original scope, including after a Quick-to-Full or model upgrade.
|
|
110
|
+
|
|
111
|
+
Preparation state, raw observations and the decision journal are private local
|
|
112
|
+
runtime records excluded from automatic Git delivery. Do not publish credentials
|
|
113
|
+
or source excerpts from logs automatically. Write a separately reviewed report when
|
|
114
|
+
publication is part of the user's goal. Once admitted, deleting or changing the
|
|
115
|
+
preparation source cannot bypass scope or accounting.
|
|
116
|
+
|
|
117
|
+
## Feedback, progress and Plan
|
|
118
|
+
|
|
119
|
+
Executor prompts recommend targeted checks first. Go still runs the complete frozen
|
|
120
|
+
acceptance against the current candidate before independent Review. This removes a
|
|
121
|
+
fixed requirement to run the full suite twice; it does not cache or remove the gate.
|
|
122
|
+
|
|
123
|
+
Feedback carries stable check IDs, expected and actual physical outcomes,
|
|
124
|
+
candidate/check-set/environment identities, a full failed-output digest and bounded
|
|
125
|
+
head/tail excerpts. `observationComplete` describes command-outcome completeness,
|
|
126
|
+
not a complete test inventory. A repeated identical failure is stalled only for
|
|
127
|
+
comparable candidate, checks and environment. A changed candidate without a trusted
|
|
128
|
+
oracle is unknown and retains the total attempt cap. File names alone do not prove
|
|
129
|
+
stagnation, and free text cannot establish a trustworthy “10 failures to 1” metric.
|
|
130
|
+
|
|
131
|
+
A Full Plan is reusable only after a confirmed terminal result, valid reloaded
|
|
132
|
+
contracts, Git persistence and worktree synchronization. Normal repairs can retain
|
|
133
|
+
that proof. Contract, verification or scheduling changes, an unknown external HEAD,
|
|
134
|
+
legacy missing proof, explicit Plan or a verification gap require planning again.
|
|
135
|
+
Speed changes the economics of feedback; it does not erase planning or quality gates.
|
|
136
|
+
|
|
137
|
+
## Optional JEV decisions
|
|
138
|
+
|
|
139
|
+
Preparation `decision` is independent of legacy `jev-shadow` and absent by default.
|
|
140
|
+
Its schema defines an explicit endpoint, fixed `jev-x.y.z` model, credential
|
|
141
|
+
environment-variable name, timeout, policy version, mode, allowed data categories,
|
|
142
|
+
thresholds and maximum calls. Credential values never enter the contract.
|
|
143
|
+
|
|
144
|
+
- Off/offline do not contact the service; shadow captures a local snapshot;
|
|
145
|
+
advisory records advice; adopt can select one approved diagnostic action.
|
|
146
|
+
Thresholds are explicit experiment parameters, not measured deployment accuracy.
|
|
147
|
+
- At least two legal choices and actual semantic uncertainty are required. One
|
|
148
|
+
request pairs applicability Noul with Choice and retains full probabilities.
|
|
149
|
+
Missing applicability, version mismatch, unauthorized fields, insufficient budget
|
|
150
|
+
or unknown execution cause abstention.
|
|
151
|
+
- Before adoption, recheck task/run/revision, contracts, candidate, observations and
|
|
152
|
+
action binding. Consume the decision and bind a unique action invocation in one
|
|
153
|
+
CAS before dispatch. JEV cannot invent commands, skip acceptance or declare success.
|
|
154
|
+
- An unknown optional request is not resent. Usage remains unknown and the task can
|
|
155
|
+
take its legal deterministic baseline. An unknown action outcome blocks. Ending,
|
|
156
|
+
cancellation and deadline expiration still reconcile the optional request read-only.
|
|
157
|
+
- Maximum calls include unknowns. Hard budgets skip optional JEV when the API cannot
|
|
158
|
+
enforce a provable request ceiling. Character estimates are not reservations.
|
|
159
|
+
|
|
160
|
+
Legacy workflow shadow captures only a local bounded summary; explicit `shadow run`
|
|
161
|
+
evaluates it later. That switch cannot adopt actions or authorize new source/log
|
|
162
|
+
sharing. Offline or shadow results do not establish a causal speedup. Compare full
|
|
163
|
+
tasks using the existing baseline, core improvements without JEV, and those same
|
|
164
|
+
improvements plus one action family.
|
|
165
|
+
|
|
166
|
+
### Offline replay and explicit advisory evaluation
|
|
167
|
+
|
|
168
|
+
The general decision capture is a `*.snapshot.json` document containing the frozen
|
|
169
|
+
`snapshot`, `policy` and source `target-hash`. It is distinct from the legacy workflow
|
|
170
|
+
shadow dataset. Review its facts and data permissions before any live evaluation.
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
flywheel decision replay --input frozen.snapshot.json --response native-response.json \
|
|
174
|
+
--endpoint "$JEV_ENDPOINT" --model jev-1.14.0 --expected-model jev-1.14.0
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Replay accepts a recorded native System One response or a fixture, forces `offline`,
|
|
178
|
+
does not read a credential, sends no HTTP and writes no journal or task state. The
|
|
179
|
+
endpoint and equal fixed model IDs identify the intended target; the version above
|
|
180
|
+
is an example and must match the response under evaluation. A bare Snapshot JSON
|
|
181
|
+
requires `--policy reviewed-policy.json`. That flag also replaces the entire captured
|
|
182
|
+
policy when used with a wrapper; omitted data permissions are not inherited.
|
|
183
|
+
|
|
184
|
+
For an explicitly authorized live experiment:
|
|
185
|
+
|
|
186
|
+
```bash
|
|
187
|
+
flywheel decision evaluate --input frozen.snapshot.json --policy reviewed-policy.json \
|
|
188
|
+
--endpoint "$JEV_ENDPOINT" --model jev-1.14.0 --expected-model jev-1.14.0 \
|
|
189
|
+
--directory ./decision-evaluation --key-env MEROUTER_API_KEY --max-calls 5
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
`evaluate` always forces `advisory`, even if the source policy says `adopt`. The
|
|
193
|
+
credential environment-variable name, fixed target, separate private journal and
|
|
194
|
+
cumulative call cap are explicit. It preserves source data-category permissions and
|
|
195
|
+
requires them to cover the snapshot. `--hard-budget` skips the optional call because
|
|
196
|
+
the native API cannot enforce a request usage ceiling. Reusing the same journal and
|
|
197
|
+
snapshot reads the original receipt; unknown requests consume the cap and are never
|
|
198
|
+
resent. The command does not execute an action or change task state.
|
|
199
|
+
|
|
200
|
+
Both commands print the baseline, effective policy, probabilities and receipt with
|
|
201
|
+
`adopted: false`. Live errors or unknown outcomes return exit code 3 with that receipt;
|
|
202
|
+
input/configuration errors return 1, cancellation before dispatch returns 143. A
|
|
203
|
+
successful offline replay is parser/fixture evidence, not a measured model success.
|
|
204
|
+
Limited task adoption still requires the separate preparation policy, data opt-in,
|
|
205
|
+
frozen concrete Go action and current-state checks described above.
|
|
206
|
+
|
|
207
|
+
## Measurement and remaining boundaries
|
|
208
|
+
|
|
209
|
+
Alongside invocation records, `metrics.jsonl` now contains `phase-start` and
|
|
210
|
+
`phase-end`. `control metrics` reports cumulative span duration separately from the
|
|
211
|
+
union of completed intervals. Parallel spans cannot be summed as wall time. Missing
|
|
212
|
+
ends, clock regressions and unmatched legacy calls remain explicit. Provider queue,
|
|
213
|
+
model and internal tool breakdown remains unknown. Investigation actions/runs use
|
|
214
|
+
the same phase API; unobserved host-preparation time is never filled with zero.
|
|
215
|
+
|
|
216
|
+
Offline checks do not establish fewer tokens, lower cost or equal final quality.
|
|
217
|
+
This change does not enable a new default model, JEV, prewarming or higher
|
|
218
|
+
concurrency. Ready queues, persistent workers and importing existing candidates
|
|
219
|
+
remain conditional on measurement and their own recovery/acceptance evidence.
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# Optional JEV shadow evaluation
|
|
2
|
+
|
|
3
|
+
[Documentation](../README.md) · English · [简体中文](../zh-CN/jev-shadow.md)
|
|
4
|
+
|
|
5
|
+
This guide covers the optional workflow-shadow experiment. It captures a bounded
|
|
6
|
+
task summary for later, explicit evaluation and does not choose the production workflow. Start with the
|
|
7
|
+
[normal workflow](workflow.md); enable shadow evaluation only when collecting
|
|
8
|
+
comparison evidence for a configured endpoint.
|
|
9
|
+
|
|
10
|
+
## Enable or disable
|
|
11
|
+
|
|
12
|
+
JEV is **off by default**. Missing `jev-shadow` in older configuration also means off: no credential lookup, network requests or shadow records. Existing Go Quick/Full selection stays in charge.
|
|
13
|
+
|
|
14
|
+
In the desktop, open **Flywheel → Configuration**, choose the global or project
|
|
15
|
+
scope, then open **JEV shadow advice**. Configuration loads automatically; review
|
|
16
|
+
the endpoint and save. Alternatively merge this section into the effective
|
|
17
|
+
complete `flywheel.config.json`:
|
|
18
|
+
|
|
19
|
+
```json
|
|
20
|
+
{
|
|
21
|
+
"jev-shadow": {
|
|
22
|
+
"enabled": false,
|
|
23
|
+
"endpoint": "https://merouter.play.one2x.ai/v1/systemone",
|
|
24
|
+
"model": "jev",
|
|
25
|
+
"expected-model": "jev-1.14.0",
|
|
26
|
+
"key-env": "MEROUTER_API_KEY",
|
|
27
|
+
"timeout-ms": 5000
|
|
28
|
+
}
|
|
29
|
+
}
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The shared template leaves the endpoint empty. Configuration stores the environment variable **name**, never the key. Local capture does not look up credentials; the process running an explicit `shadow run` must inherit the variable. Never put credentials in endpoint or model fields. Discovery selects one complete file in task → project → global → embedded order, without merging. Inspect the source with `config inspect --json`; saves affect subsequent starts. Disabling preserves previous records.
|
|
33
|
+
|
|
34
|
+
Enabling captures task goals, requirement/acceptance descriptions and planning evidence privately for later review and labeling. The task itself sends nothing to JEV. No source files, diffs, execution logs or provider conversations are read. URL/home-prefix removal and common credential-pattern rejection are best effort, not universal anonymization. Review the business summary before explicitly sending it to your configured endpoint. This setting never authorizes a new data category or automatic adoption of diagnostic decisions.
|
|
35
|
+
|
|
36
|
+
Only a new task's first ordinary run is captured, after existing workflow state is saved. It writes the bounded input and pinned target locally, with no HTTP wait, credential lookup or background goroutine. Resume, retry and single-step commands do not backfill observations. Invalid targets, rejected summaries and record failures log a warning and leave the primary workflow running. Hard-budget tasks can also capture, because capture creates no API usage. Cancellation skips capture and retains the normal task-stop behavior. Explicit Full and structural Quick exclusions remain deterministic blockers when a later experiment evaluates the snapshot.
|
|
37
|
+
|
|
38
|
+
Advice never changes Quick/Full, routing, permissions, sessions, acceptance or PR/Merge gates. Independent usage and latency records do not enter the task budget. Unknown usage stays unknown; no billing savings are implied.
|
|
39
|
+
|
|
40
|
+
## Explicit experiments
|
|
41
|
+
|
|
42
|
+
Automatic captures live under `~/.config/flywheel/shadow/<target-digest>/<input-digest>/` (0700 directories, 0600 files). Logs identify the directory. `cases.json` holds the bounded summary; `target.json` pins endpoint, model, questions and policy. Before explicit evaluation, the report shows these cases as not run. The later `shadow run` owns its lifetime and separate call budget; it adds probabilities, suggestions, actual baseline, response model and usage. An intent without a terminal remains unknown and is not replayed. A different directory represents an explicit new experiment.
|
|
43
|
+
|
|
44
|
+
With a Flywheel version that provides `shadow prepare|run|report`:
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
flywheel shadow prepare --task-dir docs/my-task > /tmp/jev-cases.json
|
|
48
|
+
# Review the summary before sending it; supply MEROUTER_API_KEY in your environment.
|
|
49
|
+
flywheel shadow run --input /tmp/jev-cases.json \
|
|
50
|
+
--directory /tmp/my-jev-experiment \
|
|
51
|
+
--endpoint https://merouter.play.one2x.ai/v1/systemone \
|
|
52
|
+
--max-calls 20 --timeout 5s
|
|
53
|
+
flywheel shadow report --input /tmp/jev-cases.json \
|
|
54
|
+
--directory /tmp/my-jev-experiment
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
These explicit commands remain usable with automatic observation disabled. They do not start or mutate a task. Preparation requires a prepared or quiescent task. Reporting is offline and needs no key. For an automatic capture, use its `cases.json` as input. Copy it elsewhere before adding independent labels (`mode`, `status`, `author`, `reason`); labels are excluded from both requests and prediction cache identity. Preserve original captures.
|
|
58
|
+
|
|
59
|
+
Datasets are limited to 100 cases and 1 MiB, requests to 64 KiB. Explicit `--max-calls` is 1–100; timeout is at most 30 seconds. A batch exceeding the new-call limit is rejected before dispatch. There are no HTTP redirects or retries; error response bodies are not logged. Commands print one JSON document on stdout, errors on stderr. Exit codes: 0 report success, 1 invalid input/runtime failure, 3 failed/unknown/not-run cases, 143 cancellation.
|
|
60
|
+
|
|
61
|
+
## Interpretation
|
|
62
|
+
|
|
63
|
+
Eight atomic Noul questions use fixed experimental thresholds: every positive condition ≥0.8 and every risk ≤0.2 yields a Quick candidate; any risk ≥0.8 yields Full; otherwise abstain. The manifest pins questions, thresholds and versions. Individual probabilities are not an overall accuracy score. Unexpected response models abstain until an operator reviews and updates the pin.
|
|
64
|
+
|
|
65
|
+
This policy is **not production calibrated**. Reports separate decision-time, prepared, replay, historical and synthetic cases, and provisional versus confirmed labels. Current historical state need not be its initial Auto decision. Missing/provisional labels do not establish a false-Quick rate. Auto already uses preparation-time contract evidence; this experiment adds usage rather than removing a classifier call that never existed.
|
|
66
|
+
|
|
67
|
+
The [six historical summaries](../evaluation/jev-historical-provisional.json) reference completed commits, with unknown actual modes and provisional labels. They test the interface and expose counterexamples, not production benefits. Real automatic workflow routing would require independently confirmed decision-time samples and measured false-Quick/abstention rates. There is no JEV Quick/Full routing switch. The separate bounded diagnostic decision policy cannot inherit this experiment's permissions or thresholds.
|
package/docs/en/model-policy.md
CHANGED
|
@@ -1,63 +1,148 @@
|
|
|
1
|
-
#
|
|
2
|
-
|
|
3
|
-
[
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
|
18
|
-
|
|
|
19
|
-
|
|
|
20
|
-
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
1
|
+
# Models, reasoning effort, and accounts
|
|
2
|
+
|
|
3
|
+
[Documentation](../README.md) · English · [简体中文](../zh-CN/model-policy.md)
|
|
4
|
+
|
|
5
|
+
Flywheel selects a model through a stage or execution-route binding. The
|
|
6
|
+
[default template](https://github.com/ckanner/flywheel/blob/main/templates/flywheel.config.example.json) is the source
|
|
7
|
+
of truth for new configurations; an existing installation keeps its own
|
|
8
|
+
configuration. These defaults are an editable baseline. The repository does not
|
|
9
|
+
establish that they are the optimum for representative tasks, or that increasing
|
|
10
|
+
reasoning effort improves every delivery.
|
|
11
|
+
|
|
12
|
+
## Default bindings
|
|
13
|
+
|
|
14
|
+
| Stage or route | Harness | Model | Access path | Effort |
|
|
15
|
+
|---|---|---|---|---|
|
|
16
|
+
| Plan | `kimi` | `kimi-code/k3.1-pro` | Kimi Code subscription through the native CLI | `high` |
|
|
17
|
+
| Execute: default | `dsh` | `deepseek-flash` | DSH `deepseek-account` | `high` |
|
|
18
|
+
| Execute: frontend | `kimi` | `kimi-code/k3.1-pro` | Kimi Code subscription through the native CLI | `high` |
|
|
19
|
+
| Execute: hard | `codex-pro` | `gpt-6.1-sol` | Codex login through the public machine API | `high` |
|
|
20
|
+
| Review | `dsh` | `gpt-6.1-sol` | DSH `chatgpt` provider | `high` |
|
|
21
|
+
| Judge | `dsh` | `gpt-6-astra` | DSH `chatgpt` provider | `high` |
|
|
22
|
+
| Intake | `dsh` | `deepseek-flash` | DSH `deepseek-account` | `low` |
|
|
23
|
+
| Garden | `kimi` | `kimi-code/k3.1-pro` | Kimi Code subscription through the native CLI | `high` |
|
|
24
|
+
|
|
25
|
+
The harness owns the agent/tool loop. A DSH `provider` selects its native LLM
|
|
26
|
+
adapter, while a `channel` names the configured account/credential path. A
|
|
27
|
+
`codex-pro` harness is not a DSH provider called `subagent_codex_pro`. Kimi uses
|
|
28
|
+
its native CLI, not ACP. Auto, Quick, and Full do not change these bindings or
|
|
29
|
+
increase effort automatically; see [Configuration](configuration.md).
|
|
30
|
+
|
|
31
|
+
Claude Code Pro is available as an [optional backend](claude-code-pro.md), with
|
|
32
|
+
no default stage or route assigned. Its profile uses an explicit native model
|
|
33
|
+
and effort rather than an implicit replacement for another subscription.
|
|
34
|
+
|
|
35
|
+
## Account selection
|
|
36
|
+
|
|
37
|
+
The default and `subscriptions` preset use `deepseek-account` with
|
|
38
|
+
`deepseek-flash`. Standard DSH desktop and SDK profiles under the same DSH home
|
|
39
|
+
can share account authorization without an additional API key. Custom profiles
|
|
40
|
+
must load the account service and adapter. Independent SDK calls still supply
|
|
41
|
+
an explicit provider/model; changing the desktop chat model does not update
|
|
42
|
+
Flywheel's bindings.
|
|
43
|
+
|
|
44
|
+
`deepseek-official` is a separate API-key provider, selected only by explicit
|
|
45
|
+
configuration. The default preset does not switch payment channels after an
|
|
46
|
+
authentication or quota failure. User-configured fallback chains retain their
|
|
47
|
+
own policy. ChatGPT and Kimi subscriptions likewise require the corresponding
|
|
48
|
+
installed adapter or CLI and its supported login flow.
|
|
49
|
+
|
|
50
|
+
Upgrades preserve existing user files. To adopt the subscription preset, inspect
|
|
51
|
+
the effective complete configuration, preview the migration, then explicitly
|
|
52
|
+
apply it as described in [Configuration](configuration.md). Restart the engine
|
|
53
|
+
afterward; neither installing packages nor changing a desktop chat model
|
|
54
|
+
migrates a running task.
|
|
55
|
+
|
|
56
|
+
## Reasoning effort is backend-specific
|
|
57
|
+
|
|
58
|
+
The following capabilities were checked for the documented integrations on
|
|
59
|
+
**2026-11-05**. They are compatibility evidence for those snapshots, not a live
|
|
60
|
+
catalog or an account-entitlement check.
|
|
61
|
+
|
|
62
|
+
| Checked integration | Accepted request values | Notes |
|
|
31
63
|
|---|---|---|
|
|
32
|
-
|
|
|
33
|
-
|
|
|
34
|
-
|
|
|
35
|
-
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
##
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
+
| GPT-6.1 Sol / Codex Pro 0.9.0 | `low`, `medium`, `high`, `xhigh`, `max` | The machine API forwards the selected effort |
|
|
65
|
+
| Sol and Astra / DSH ChatGPT provider | `low`, `medium`, `high`, `xhigh`, `max` | The provider serializes `reasoning.effort` |
|
|
66
|
+
| Kimi K3.1 Pro / native CLI | `low`, `high`, `max` | Thinking cannot be disabled; compatibility aliases `xhigh`, `max`, and `ultra` resolve to native `max` |
|
|
67
|
+
| DeepSeek Flash / checked DSH adapter | `off`, `low`, `high`, `max` | This adapter rejects `medium` and `xhigh` |
|
|
68
|
+
|
|
69
|
+
Names do not form a common scale across providers. In particular, a generic
|
|
70
|
+
API's compatibility mapping does not override the DSH adapter's own input
|
|
71
|
+
validation. Use `max`, not `xhigh`, when evaluating Flash's higher supported
|
|
72
|
+
level on this path. The adapter must reject an unsupported selection rather
|
|
73
|
+
than silently choose another model or effort.
|
|
74
|
+
|
|
75
|
+
Compatibility references: [Kimi model documentation](https://www.kimi.com/code/docs/en/kimi-code/models),
|
|
76
|
+
[DeepSeek thinking](https://api-docs.deepseek.com/guides/thinking_mode/),
|
|
77
|
+
[Sol model reference](https://developers.openai.com/api/docs/models/gpt-6.1-sol),
|
|
78
|
+
[Astra model reference](https://developers.openai.com/api/docs/models/gpt-6-astra),
|
|
79
|
+
[DSH DeepSeek configuration](https://github.com/deepseek-ai/deepseek-harness/blob/b830ec950faba0a46dda564be3e2d854afed94f1/packages/llm/llm-deepseek/src/config.ts),
|
|
80
|
+
and [serialization](https://github.com/deepseek-ai/deepseek-harness/blob/b830ec950faba0a46dda564be3e2d854afed94f1/packages/llm/llm-deepseek/src/serialize.ts).
|
|
81
|
+
The checked ChatGPT provider is 0.1.17 at `388868f91bcdc05795fb94d21b868ded6d7219c9`:
|
|
82
|
+
[model catalog](https://github.com/ckanner/dsh-plugin-chatgpt/blob/388868f91bcdc05795fb94d21b868ded6d7219c9/src/models/served.ts)
|
|
83
|
+
and [request serialization](https://github.com/ckanner/dsh-plugin-chatgpt/blob/388868f91bcdc05795fb94d21b868ded6d7219c9/src/convert/request.ts).
|
|
84
|
+
|
|
85
|
+
## Choose and evaluate a baseline
|
|
86
|
+
|
|
87
|
+
Seven of the eight default bindings use `high`; Intake uses `low`. Keeping a
|
|
88
|
+
stable starting point makes comparisons attributable to a particular stage.
|
|
89
|
+
The following candidates are engineering judgments based on stage
|
|
90
|
+
responsibilities, not measured Flywheel quality results.
|
|
91
|
+
|
|
92
|
+
| Stage | Candidate comparison |
|
|
93
|
+
|---|---|
|
|
94
|
+
| Intake | Compare `high` for ambiguous input, merging, or classification |
|
|
95
|
+
| Plan | Compare `max` for difficult architecture, dependencies, or acceptance design |
|
|
96
|
+
| Execute: default | Compare Flash `max` for difficult implementation or root-cause analysis |
|
|
97
|
+
| Execute: frontend | Compare Kimi `max` for complex interaction state or cross-component consistency |
|
|
98
|
+
| Execute: hard | Compare Sol `xhigh` first; evaluate `max` on the hardest tasks |
|
|
99
|
+
| Review | Compare `xhigh` for concurrency, consistency, or permission reviews |
|
|
100
|
+
| Judge | Compare Astra `xhigh` for high-impact milestone acceptance, then `max` where justified |
|
|
101
|
+
| Garden | Compare `low` for routine organization and `max` for difficult cross-task synthesis |
|
|
102
|
+
|
|
103
|
+
Change `thinking` in the effective complete configuration. For example, change
|
|
104
|
+
`routes.hard.thinking` to affect only hard execution, or
|
|
105
|
+
`loops.judge.thinking` to affect only Judge. A stage override avoids changing all
|
|
106
|
+
consumers of a shared profile. Do not create an incomplete task configuration
|
|
107
|
+
containing only these fields: configuration discovery selects one file rather
|
|
108
|
+
than merging files.
|
|
109
|
+
|
|
110
|
+
Hold the model, harness, context, task, permissions, and acceptance conditions
|
|
111
|
+
fixed. Measure delivery success, first-pass acceptance, missed defects and false
|
|
112
|
+
alarms, total time including rework, and input/output/reasoning/cache usage.
|
|
113
|
+
Record observable subscription quota separately and preserve unknown values.
|
|
114
|
+
API prices do not directly determine subscription task capacity. The
|
|
115
|
+
[session evaluation guide](session-evaluation.md) describes the repository's
|
|
116
|
+
paired experiment tooling; effort comparisons need their own fixed conditions.
|
|
117
|
+
|
|
118
|
+
## Bounded model and effort experiments
|
|
119
|
+
|
|
120
|
+
The optional [execution policy](configuration.md#bounded-execution-policy-opt-in)
|
|
121
|
+
declares complete allowed bindings per route, including account, effort,
|
|
122
|
+
permission and history mode. Auto/Quick/Full do not enable it. A confirmed
|
|
123
|
+
repeated L1 stall may advance the declared experiment order within existing
|
|
124
|
+
attempt/token budgets and an explicit switch limit. This deterministic rule
|
|
125
|
+
does not prove the next model is better. Channel errors use separately declared
|
|
126
|
+
fallbacks; neither mechanism may select an undeclared model or payment channel.
|
|
127
|
+
|
|
128
|
+
Effort changes cold-start the role and record complete before/after bindings.
|
|
129
|
+
Comparisons must include rereading, startup and actual known/unknown usage and
|
|
130
|
+
cost. Offline tests run real engine repair and recovery paths with controlled
|
|
131
|
+
driver results; they do not establish live model quality or speed. Use matched
|
|
132
|
+
real tasks with frozen oracles before changing defaults or claiming savings.
|
|
133
|
+
|
|
134
|
+
## Interpret configuration and quality evidence
|
|
135
|
+
|
|
136
|
+
Source and offline protocol tests establish field transmission and error
|
|
137
|
+
behavior. SDK or machine-API acceptance establishes acceptance at that interface.
|
|
138
|
+
Independent effective-effort evidence and better delivery quality require
|
|
139
|
+
backend evidence or actual task measurements.
|
|
140
|
+
|
|
141
|
+
DSH `initialize` validates configuration but returns only server information.
|
|
142
|
+
Flywheel's `ready.reasoningEffort` echoes the request after initialization;
|
|
143
|
+
it is not independent model-server attestation. Codex Pro separates
|
|
144
|
+
`requestedEffort`, `accepted`, and an unknown `effectiveEffort:null`.
|
|
145
|
+
See the [DSH SDK server](https://github.com/deepseek-ai/deepseek-harness/blob/b830ec950faba0a46dda564be3e2d854afed94f1/packages/sdk/server/src/server.ts)
|
|
146
|
+
and [Codex Pro machine protocol](https://github.com/ckanner/dsh-subagent-codex-pro/blob/37da3d75de32b809c4fb1a631ccc15191f3d87f9/docs/machine.md).
|
|
147
|
+
A no-inference doctor cannot prove account balance, live inference, or the
|
|
148
|
+
benefit of a higher effort setting.
|