dsh-autoctx 0.0.0-stage → 1.2.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +9 -0
- package/README.en.md +203 -0
- package/README.md +202 -2
- package/briefs.js +391 -0
- package/cachespace.js +49 -0
- package/client.js +835 -0
- package/cordis.patch.yml +6 -0
- package/decisions.js +246 -0
- package/graph.js +619 -0
- package/index.js +3163 -0
- package/learning.js +165 -0
- package/ledger.js +161 -0
- package/models.js +131 -0
- package/models.json +225 -0
- package/off.patch.yml +5 -0
- package/package.json +58 -4
- package/routing.js +85 -0
- package/segments.js +186 -0
- package/stats.js +77 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Zhiwei Wei
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
|
6
|
+
|
|
7
|
+
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
|
8
|
+
|
|
9
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
package/README.en.md
ADDED
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
# dsh-autoctx
|
|
2
|
+
|
|
3
|
+
Context and subagent decision plugin for dsh, version **1.2.5**. Jev scores options and makes structured choices; the plugin implements them through native dsh sessions, tools, subagents and compaction. The executor model still does all the coding.
|
|
4
|
+
|
|
5
|
+
[中文](https://github.com/ZhiweiWei-NAMI/dsh-autoctx/blob/main/README.md) · [Source](https://github.com/ZhiweiWei-NAMI/dsh-autoctx) · MIT
|
|
6
|
+
|
|
7
|
+
## Environment and accounts
|
|
8
|
+
|
|
9
|
+
Unit tests pass on **dsh `0.2.0-rc.2` and `0.2.1-rc.2`**, using Node 22; the benchmark below ran on `0.2.0-rc.2`, and real smoke runs were verified on `0.2.1-rc.2`. Peer dependencies for the `@deepseek-ai/dsh-*` packages accept `^0.1.5-rc.1 || ^0.2.0-rc.1 || ^0.2.1-rc.1`. 1.2.5 decides thinking/effort at decision points per trigger, using evidence and a cost estimate per option, as detailed under “How it works”; compatibility with dsh 0.2.1's subagent start API (`startActivation`) and 0.2.0 is kept. The npm package is `dsh-autoctx`, its entry is `index.js`, and its web client is `./client.js`.
|
|
10
|
+
|
|
11
|
+
You must provide:
|
|
12
|
+
|
|
13
|
+
- **An executor model connection** in the dsh profile's settings/provider. The plugin does not provide model access.
|
|
14
|
+
- **A Jev API key**: obtain a workspace key from [opencode.ai](https://opencode.ai) and set `apiKey`. This key is sent to `jevEndpoint`, which defaults to `https://opencode.ai/inference/v1/systemone`.
|
|
15
|
+
- **Optional extra model routes**: if reviewer/advisor roles use other models, configure `modelRoutes` as model id → dsh provider name. Paid routes also require `allowPaidModels: true`.
|
|
16
|
+
|
|
17
|
+
Without `apiKey`, the plugin makes no network call, the session runs in native mode, and it records `reason: 'no-api-key'`.
|
|
18
|
+
|
|
19
|
+
## Install and configure
|
|
20
|
+
|
|
21
|
+
1. Create the profile on first use. This command boots it; stop it with Ctrl+C before installing.
|
|
22
|
+
|
|
23
|
+
```sh
|
|
24
|
+
dsh --profile autoctx --from-default-profile web
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
For a headless profile, replace the template with `headless` in the initial creation command. Start your custom profile with `dsh --profile autoctx ...`.
|
|
28
|
+
|
|
29
|
+
2. Install from npm, or choose the GitHub release tag source.
|
|
30
|
+
|
|
31
|
+
```sh
|
|
32
|
+
dsh plugin --profile autoctx add dsh-autoctx
|
|
33
|
+
# Or use the release tag
|
|
34
|
+
dsh plugin --profile autoctx add github:ZhiweiWei-NAMI/dsh-autoctx#v1.2.5
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
`dsh plugin --profile <name> <pnpm-args...>` passes arguments to pnpm. Profiles live in `$DSH_HOME/profiles/<name>`. The package's declared dsh bundle patch (`cordis.patch.yml`) mounts the plugin and disables native `compaction-basic`; no duplicate manual mount is needed.
|
|
38
|
+
|
|
39
|
+
3. Configure the executor model connection in dsh itself, and add this minimal configuration to **the profile's own** `$DSH_HOME/profiles/autoctx/cordis.patch.yml`, not the home-level patch.
|
|
40
|
+
|
|
41
|
+
```yaml
|
|
42
|
+
- id: dsh-autoctx
|
|
43
|
+
config:
|
|
44
|
+
apiKey: <your opencode workspace key>
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
4. Start the web app.
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
dsh --profile autoctx
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Disable the plugin and restore native compaction for one launch:
|
|
54
|
+
|
|
55
|
+
```sh
|
|
56
|
+
dsh --profile autoctx --patch "$DSH_HOME/profiles/autoctx/node_modules/dsh-autoctx/off.patch.yml"
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Upgrade to the latest version, or pin this version:
|
|
60
|
+
|
|
61
|
+
```sh
|
|
62
|
+
dsh plugin --profile autoctx add dsh-autoctx@latest
|
|
63
|
+
# Or pin the version
|
|
64
|
+
dsh plugin --profile autoctx add dsh-autoctx@1.2.5
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Before uninstalling, delete any `- id: dsh-autoctx` override you added to the profile's `cordis.patch.yml`, then run:
|
|
68
|
+
|
|
69
|
+
```sh
|
|
70
|
+
dsh plugin --profile autoctx remove dsh-autoctx
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Removing the bundle layer restores native compaction.
|
|
74
|
+
|
|
75
|
+
## Settings and defaults
|
|
76
|
+
|
|
77
|
+
In the web UI, open bottom-left **Settings → DSH AutoCTX**, or **Plugins → dsh-autoctx → Configure**. Fields include mode, learning, context selection, adaptive thinking effort, Jev endpoint, model routes and ceiling ratio. A composer chip shows context pressure and decisions; its card shows the schedule, thinking policy and learning ledger, including pending proposals in manual mode. `apiKey` is a secret field and is not written to logs or record files.
|
|
78
|
+
|
|
79
|
+
| Key | Default and meaning |
|
|
80
|
+
| --- | --- |
|
|
81
|
+
| `mode` | `auto`; choices: `auto`, `single`, `native` |
|
|
82
|
+
| `apiKey` | Unset; native mode when unset |
|
|
83
|
+
| `jevModel` | `jev-1.13`, billed; `jev-1.13-free` is available but rate limited |
|
|
84
|
+
| `jevEndpoint` | `https://opencode.ai/inference/v1/systemone` |
|
|
85
|
+
| `modelRoutes` | `{}`; model id → dsh provider name |
|
|
86
|
+
| `allowPaidModels` | `false`; whether to allow paid routes |
|
|
87
|
+
| `reviewerModel` | `same`; fork review with the executor's model; `auto` lets Jev choose, or specify a model id |
|
|
88
|
+
| `reviewBeforeFinish` | `true`; one independent review child before a root turn that changed files finishes |
|
|
89
|
+
| `reviewMaxRequests` | `6`; review request cap |
|
|
90
|
+
| `advisorModel` | `off`; choices: `off`, `auto`, or a model id |
|
|
91
|
+
| `advisorMaxConsults` | `1`; advisor consultation cap |
|
|
92
|
+
| `advisorStallSteps` | `6`; stall step threshold |
|
|
93
|
+
| `adaptiveEffort` | `high`; effort used when Jev's thinking policy calls for it; `off` keeps the default effort |
|
|
94
|
+
| `effortSwitchFreeTokens` | `4096`; a decision-point thinking switch whose estimated uncached re-read is at most this proceeds without asking Jev |
|
|
95
|
+
| `maxEffortDecisions` | `6`; separate Jev effort-decision calls per root task; never spends `maxDecisions` |
|
|
96
|
+
| `effortPhaseRequests` | `4`; maximum high-effort requests in a thinking phase; a new trigger renews the window |
|
|
97
|
+
| `cacheTtlSeconds` | `300`; assumed cache-reuse lifetime used in runtime estimates; the provider TTL is unverified |
|
|
98
|
+
| `learning` | `frozen`; choices: `auto`, `manual`, `frozen` |
|
|
99
|
+
| `contextSelection` | `jev`; choices: `jev`, `summary` |
|
|
100
|
+
| `maxDecisions` | `12`; Jev routing/context decision calls per root task |
|
|
101
|
+
| `maxChildren` | `2`; child count cap |
|
|
102
|
+
| `maxConcurrent` | `2`; concurrent child count cap |
|
|
103
|
+
| `maxChildRequests` | `16`; child request cap |
|
|
104
|
+
| `maxChildTurnRequests` | `8`; child request cap per turn |
|
|
105
|
+
| `pressureTokens` | `64000` tokens; context pressure threshold |
|
|
106
|
+
| `ceilingRatio` | `1.5`; context must shrink at `pressureTokens × ceilingRatio` |
|
|
107
|
+
| `retainTokens` | `8000` tokens; retained-token setting |
|
|
108
|
+
|
|
109
|
+
## How it works
|
|
110
|
+
|
|
111
|
+
**Decisions.** A plain single agent keeps one growing context; native subagents are chosen by the executor through spawn/fork, and native compaction summarizes old context. AutoCTX asks Jev for structured decisions at specific points. Jev is a scoring/choice model; it does not write plans or code. It decides whether a proposed delegation is worth a child and which route to use; gives a task tag and thinking policy at turn start; selects evidence under context pressure and chooses whether to replace context now or later, weighing the cost of re-reading uncached content against smaller later requests; and selects reviewer/advisor models when those roles are set to `auto`.
|
|
112
|
+
|
|
113
|
+
Thinking policies are `off`, `decision_points`, or `always`; decision points are turn start, test failure, review/advisor report and stall. In 1.2.5, each trigger supplies evidence such as a failing-test output excerpt, diff hunk, review/advisor report text or task head, with an estimated cost per option at catalog prices: `skip` stays in the current cache space; `pulse` makes one high-effort request; `phase` keeps high effort until the trigger is resolved (test passes, report acted on, progress) or `effortPhaseRequests` requests. Cheap switches (estimated uncached re-read ≤ `effortSwitchFreeTokens`) and switches right after a context replacement proceed without asking Jev. Effort questions use their own `maxEffortDecisions` budget and never spend the 12 routing/context decisions in `maxDecisions`. Cache reuse is estimated from exact request prefixes, with no rewriting, reordering or dropping of context for cache; these runtime estimates are theoretical, not actual bills.
|
|
114
|
+
|
|
115
|
+
A transient Jev gateway/network failure is retried once, then Jev pauses for 60 seconds. Authentication, billing or HTTP 429 stops Jev for the task, and local fallbacks take over: native compaction and no delegation.
|
|
116
|
+
|
|
117
|
+
**Context selection.** With `contextSelection: jev`, when context passes `pressureTokens`, the old region is split into evidence pieces—test-failure sections, diff hunks, code by function/class and paragraphs—rather than whole outputs. Jev scores how much each piece adds given the stated open problem, such as which test still fails, and the content kept regardless. Locally, duplicates keep one copy; a kept report pulls in the definition it refers to; pieces that add little are archived as one line each. At `pressureTokens × ceilingRatio`, the session must shrink, using native compaction if needed. `summary` lets the executor summarize instead.
|
|
118
|
+
|
|
119
|
+
**Reviewer.** Before a root turn that changed files finishes, one review child checks the actual change. Its brief starts with the workspace git diff and code around each changed hunk, then the issue and an evidence ledger. `DEFECTS` reports go back to the executor; `LGTM` ends the turn.
|
|
120
|
+
|
|
121
|
+
**Advisor.** Off by default. After repeated failing edit-and-rerun attempts on the same test, or a stall, a model with no tools receives a specific question, the raw failure excerpt, the diff, surrounding code and previous attempts.
|
|
122
|
+
|
|
123
|
+
**Learning.** The ledger is keyed by workspace path and stored in dsh storage `dsh_auto`. After each completed root turn, Jev first judges whether each used (model, role)'s specific claims can be checked against the later edits/test runs they name, then judges their quality. Overall success or the executor ignoring advice is not evidence. Unassessable claims change neither weight nor sample count. Native thumbs up/down on answers is also evidence.
|
|
124
|
+
|
|
125
|
+
- `frozen` (default): records evidence only; weights never change.
|
|
126
|
+
- `auto` (BETA, off by default): applies bounded updates automatically.
|
|
127
|
+
- `manual`: collects proposals into one approval card.
|
|
128
|
+
|
|
129
|
+
Automatic learning is expected to maintain quality while lowering cost over time, but that has not been demonstrated; auto did not outperform frozen in this benchmark.
|
|
130
|
+
|
|
131
|
+
## Extra calls and cost
|
|
132
|
+
|
|
133
|
+
Jev calls, reviewer/advisor children and their executor-model calls, and the turn-end learning call all cost money on your own API accounts. These are not free services. The model table in `models.json` lists catalog prices for `jev-1.13` at **$0.042 per 1 million input tokens, $0 for output tokens**; `jev-1.13-free` is rate limited. Benchmark costs below are theoretical estimates at catalog prices, not actual bills (defined in the benchmark section). Over the 72 cases, 1.2.5 makes 368 Jev calls versus 233 for 1.2.3 frozen; these calls also count toward a task's request limit.
|
|
134
|
+
|
|
135
|
+
## Benchmark results and limitations
|
|
136
|
+
|
|
137
|
+
These results describe this run, not general superiority. Every arm uses the same **72 graded cases = 12 fixed SWE-bench Verified pytest tasks (`bench/tasks.json`) × 3 executor models × 2 runs per model**. Executors are `deepseek-v4.1-flash`, `glm-5.3-flash` and `deepseek-v4-flash`. The official SWE-bench harness **v4.1.0** decides resolved; a finished run or HTTP 200 is not a pass.
|
|
138
|
+
|
|
139
|
+
Shared limits are **600 seconds per task, 64 model requests per task (Jev included), case parallelism 3, at most 2 children / 2 concurrent children**. AutoCTX and H used a $0.10 per-case spend cap; B and C used the harness default $0.50. No case in any arm exceeded $0.066 (theoretical, at catalog prices), so the spend cap never bound.
|
|
140
|
+
|
|
141
|
+
| Arm | Configuration |
|
|
142
|
+
| --- | --- |
|
|
143
|
+
| B | Single agent; no subagent tools, native compaction |
|
|
144
|
+
| C | Native spawn/fork with the executor model only; native compaction, no reviewer/advisor/extra models |
|
|
145
|
+
| H | Native heterogeneous subagents; one native subagent tool per (model, effort) over the executor plus 3 extra models, chosen by the executor |
|
|
146
|
+
| AutoCTX 1.2.5 frozen | New release; reviewer `auto`, advisor `auto`, the same 3 extra models, `jev-1.13`; frozen learning |
|
|
147
|
+
| AutoCTX 1.2.3 frozen / auto | Previous release, same harness; reviewer `auto`, advisor `auto`, the same 3 extra models, `jev-1.13`; frozen and auto learning respectively; auto carries the ledger across cases within a run |
|
|
148
|
+
|
|
149
|
+
The 3 extra models are `glm-5.3-flash` (use `deepseek-v4.1-flash` when the executor is glm), `deepseek-v4-pro` and `claude-haiku-5-5`. 1.2.5 ran on dsh `0.2.0-rc.2`, the same host version as 1.2.3/B/C/H. B/C/H do not load the plugin and were run before 1.2.3. B/C used the same harness entry as AutoCTX; H run 1 used an earlier harness snapshot (its `run.py` differs) with the same tasks and shared limits above.
|
|
150
|
+
|
|
151
|
+
**Theoretical cost basis.** All cost and cache figures below are **theoretical estimates at catalog prices, not actual bills**. Provider-reported cache hits vary with routing, accounts and eviction, so all arms are compared on the prompt cache they could reuse given exactly what they sent, with identical assumptions and prices: a request reuses the longest prefix it shares with an earlier request of the same task in the same cache space (model + thinking/effort config [+ tools]); no expiry within a task. Uncached input uses the catalog input price (cache-write price where listed), reused input the cache-read price, and output the output price; Jev is $0.042/M input, output free; executor, child and Jev costs are all included. This benchmark's no-expiry-within-task assumption differs from the runtime lifetime assumption in `cacheTtlSeconds`.
|
|
152
|
+
|
|
153
|
+
Two measurement methods: `exact` uses per-message prefix hashes recorded by the bench observer (AutoCTX 1.2.x runs). B/C/H were recorded before prefix hashes existed, so they use a `log` reconstruction (append-only growth per session; an input drop marks a context rewrite). Validation by re-running B and C on 12 tasks with `deepseek-v4.1-flash` and prefix hashes (24 cases): `log` gave 2.4% (B) and 3.1% (C) lower cost than `exact`; on AutoCTX it gives about 11% lower cost, because context replacement changes earlier prompt content without necessarily shrinking it. The baseline theoretical figures below are therefore probably ~3% low, and the AutoCTX premium is slightly overstated rather than understated.
|
|
154
|
+
|
|
155
|
+
Costs and cache rates in this table are theoretical estimates, not actual bills.
|
|
156
|
+
|
|
157
|
+
| Metric (72 cases per arm) | AutoCTX 1.2.5 frozen | AutoCTX 1.2.3 frozen | AutoCTX 1.2.3 auto | B | C | H |
|
|
158
|
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
159
|
+
| Resolved / 72 | 53/72 | 52/72 | 49/72 | 39/72 | 44/72 | 42/72 |
|
|
160
|
+
| Paired wins/losses of 1.2.5 frozen vs this arm (exact McNemar p) | — | 6/5 (p=1.0) | — | 15/1 (p=0.001) | 12/3 (p=0.035) | 13/2 (p=0.007) |
|
|
161
|
+
| Theoretical cost, total USD (method) | $1.738 (exact) | $1.810 (exact) | $1.603 (exact) | $0.908 (log) | $0.975 (log) | $1.027 (log) |
|
|
162
|
+
| Of which theoretical cost: executor / children / Jev (USD) | 1.383 / 0.273 / 0.082 | 1.451 / 0.284 / 0.076 | 1.321 / 0.214 / 0.068 | 0.805 / 0.102 / 0 | 0.844 / 0.131 / 0 | 0.867 / 0.160 / 0 |
|
|
163
|
+
| Theoretical cost per resolved case | $0.0328 | $0.0348 | $0.0327 | $0.0233 | $0.0222 | $0.0244 |
|
|
164
|
+
| Theoretical executor cache rate | 87.0% | 85.4% | 86.5% | 94.8% | 95.1% | 95.1% |
|
|
165
|
+
| Wall time sum (seconds) | 17580 | 18060 | 17670 | 17454 | 16520 | 17076 |
|
|
166
|
+
| Jev calls ok/all | 367/368 | 233/233 | 260/260 | — | — | — |
|
|
167
|
+
|
|
168
|
+
With the `log` method applied to every arm, the theoretical total is $1.547 for AutoCTX 1.2.5 frozen versus B $0.908, C $0.975 and H $1.027.
|
|
169
|
+
|
|
170
|
+
AutoCTX resolves more cases than B, C and H on this sample and costs more; on theoretical cost, 1.2.5 frozen is about 70–80% more than C overall and ~48% more per resolved case. Comparing 1.2.5 with 1.2.3 frozen, lane-matched and `exact`: 53 vs 52 resolved (6 wins / 5 losses, no quality difference); theoretical total cost is down 4.0% ($1.738 vs $1.810), theoretical executor cache rate is 87.0% vs 85.4%, and theoretical Jev cost is up $0.006.
|
|
171
|
+
|
|
172
|
+
1.2.5 effort decisions across 72 cases: 201 cheap (no Jev call), 86 skip, 51 phase, 9 pulse, 11 after context replacement, and 11 fell back after the effort budget was spent; 19 cases reached the 64-request limit (1.2.3: 18). The limit counts Jev calls, so AutoCTX reaches it with fewer executor requests than the baselines: 10 of those 19 cases were unresolved, and Jev made 77 of their 640 requests (12%). B/C/H also hit the limit (17/20/16 cases), with executor requests only. This benchmark keeps the shared limit; a rerun excluding Jev from it has not been done.
|
|
173
|
+
|
|
174
|
+
| Executor (24 cases per model per arm; resolved / 24) | AutoCTX 1.2.5 frozen | AutoCTX 1.2.3 frozen | AutoCTX 1.2.3 auto | B | C | H |
|
|
175
|
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
176
|
+
| deepseek-v4.1-flash | 18/24 | 17/24 | 16/24 | 11/24 | 15/24 | 14/24 |
|
|
177
|
+
| glm-5.3-flash | 17/24 | 18/24 | 18/24 | 17/24 | 18/24 | 18/24 |
|
|
178
|
+
| deepseek-v4-flash | 18/24 | 17/24 | 15/24 | 11/24 | 11/24 | 10/24 |
|
|
179
|
+
|
|
180
|
+
In the auto run, 49 completed turns reached learning: 35/49 were judged unassessable and left weights unchanged; 14/49 applied small updates. Auto did not outperform frozen here.
|
|
181
|
+
|
|
182
|
+
Only these 12 pytest tasks and 3 flash executors were measured; between repeats, resolved counts vary by about ±2 cases per 12-task run. AutoCTX costs more than the baselines here on the theoretical cost basis; auto learning remains beta and off by default. There is also a host limitation: a fork created mid-turn is seeded only with completed turns, so the review brief must carry the current turn's evidence itself.
|
|
183
|
+
|
|
184
|
+
## Reproduce
|
|
185
|
+
|
|
186
|
+
From the repository, prepare the prerequisites in [bench/README.md](https://github.com/ZhiweiWei-NAMI/dsh-autoctx/blob/main/bench/README.md), including Docker and SWE-bench images, a model gateway and `JEV_API_KEY`, then run:
|
|
187
|
+
|
|
188
|
+
```sh
|
|
189
|
+
python3 bench/run.py --model deepseek-v4.1-flash --arms A --tasks bench/tasks.json --out <dir> --plugin . --parallel 3 \
|
|
190
|
+
--extra-models glm-5.3-flash,deepseek-v4-pro,claude-haiku-5-5 --reviewer auto --advisor auto --max-usd 0.10 \
|
|
191
|
+
--jev-model jev-1.13 --learning frozen # or --learning auto
|
|
192
|
+
python3 bench/run.py --model deepseek-v4.1-flash --arms B,C --tasks bench/tasks.json --out <dir> --plugin . --parallel 3
|
|
193
|
+
python3 bench/run.py --model deepseek-v4.1-flash --arms H --tasks bench/tasks.json --out <dir> --plugin . --parallel 3 \
|
|
194
|
+
--extra-models glm-5.3-flash,deepseek-v4-pro,claude-haiku-5-5 --max-usd 0.10
|
|
195
|
+
python3 bench/grade.py <dir> --parallel 3
|
|
196
|
+
python3 bench/theoretical_cost.py "A=<dir>/*/A" "B=<dir>/*/B" "C=<dir>/*/C"
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
Replace `<dir>` with an output directory. A is AutoCTX; use separate directories for frozen and auto. Run for all 3 executor models, with 2 runs per model and separate directories for independent runs. For `glm-5.3-flash`, change `--extra-models` to `deepseek-v4.1-flash,deepseek-v4-pro,claude-haiku-5-5`; `deepseek-v4-flash` uses the extra models shown above. Re-running the same command resumes unfinished cases. `--max-usd 0.10` is the bench observer's $0.10 per-case spend cap. `bench/theoretical_cost.py` estimates cost and cache reuse with the method above; its outputs are theoretical estimates, not actual bills.
|
|
200
|
+
|
|
201
|
+
## Upgrading from older versions
|
|
202
|
+
|
|
203
|
+
Install 1.2.5 using the upgrade steps above and check profile overrides against the current configuration table. Do not carry over the old README's default model or installation tag: the current default is billed `jev-1.13`, and the release tag is `v1.2.5`.
|
package/README.md
CHANGED
|
@@ -1,3 +1,203 @@
|
|
|
1
|
-
#
|
|
1
|
+
# dsh-autoctx
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
面向 dsh 的上下文与子代理决策插件,版本 **1.2.5**。Jev 负责评分和结构化选择,插件通过 dsh 原生会话、工具、子代理和压缩执行决策;编码仍由执行模型完成。
|
|
4
|
+
|
|
5
|
+
[English](https://github.com/ZhiweiWei-NAMI/dsh-autoctx/blob/main/README.en.md) · [源码](https://github.com/ZhiweiWei-NAMI/dsh-autoctx) · MIT
|
|
6
|
+
|
|
7
|
+
## 环境与所需账号
|
|
8
|
+
|
|
9
|
+
单元测试通过的宿主为 **dsh `0.2.0-rc.2` 与 `0.2.1-rc.2`**,使用 Node 22;下方基准运行于 `0.2.0-rc.2`,真实 smoke 运行已在 `0.2.1-rc.2` 验证。`@deepseek-ai/dsh-*` 包的 peer dependency 范围为 `^0.1.5-rc.1 || ^0.2.0-rc.1 || ^0.2.1-rc.1`。1.2.5 按每次触发的证据和各选项成本估算决定决策点的思考强度,细节见「工作方式」;保留对 dsh 0.2.1 子代理启动接口(`startActivation`)及 0.2.0 的兼容。npm 包名为 `dsh-autoctx`,入口为 `index.js`,Web 客户端为 `./client.js`。
|
|
10
|
+
|
|
11
|
+
你需要提供:
|
|
12
|
+
|
|
13
|
+
- **执行模型连接**:在 dsh profile 的设置/provider 中配置;插件不提供模型访问权限。
|
|
14
|
+
- **Jev API key**:在 [opencode.ai](https://opencode.ai) 获取 workspace key,填写 `apiKey`。该 key 发送至 `jevEndpoint`,默认地址为 `https://opencode.ai/inference/v1/systemone`。
|
|
15
|
+
- **可选的其他模型路由**:若 reviewer/advisor 使用其他模型,在 `modelRoutes` 中配置「模型 id → dsh provider 名称」;付费路由还需要 `allowPaidModels: true`。
|
|
16
|
+
|
|
17
|
+
未配置 `apiKey` 时,插件不发起网络调用,会话按原生模式运行,并记录 `reason: 'no-api-key'`。
|
|
18
|
+
|
|
19
|
+
## 安装与配置
|
|
20
|
+
|
|
21
|
+
1. 首次创建 profile。此命令会启动它;用 Ctrl+C 停止后继续安装。
|
|
22
|
+
|
|
23
|
+
```sh
|
|
24
|
+
dsh --profile autoctx --from-default-profile web
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
需要 headless profile 时,首次创建命令中的模板改为 `headless`。自定义 profile 启动时使用 `dsh --profile autoctx ...`。
|
|
28
|
+
|
|
29
|
+
2. 从 npm 安装,或选择 GitHub 发布标签来源。
|
|
30
|
+
|
|
31
|
+
```sh
|
|
32
|
+
dsh plugin --profile autoctx add dsh-autoctx
|
|
33
|
+
# 或使用发布标签
|
|
34
|
+
dsh plugin --profile autoctx add github:ZhiweiWei-NAMI/dsh-autoctx#v1.2.5
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
`dsh plugin --profile <name> <pnpm-args...>` 把参数传给 pnpm。Profile 位于 `$DSH_HOME/profiles/<name>`。包声明的 dsh bundle patch(`cordis.patch.yml`)会挂载插件并停用原生 `compaction-basic`,无需手工重复挂载。
|
|
38
|
+
|
|
39
|
+
3. 配置 dsh 自身的执行模型连接,并在 **profile 自己的** `$DSH_HOME/profiles/autoctx/cordis.patch.yml` 中加入以下最小配置,不要写到 home 级 patch。
|
|
40
|
+
|
|
41
|
+
```yaml
|
|
42
|
+
- id: dsh-autoctx
|
|
43
|
+
config:
|
|
44
|
+
apiKey: <your opencode workspace key>
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
4. 启动 Web 应用。
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
dsh --profile autoctx
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
仅一次启动停用插件并恢复原生压缩:
|
|
54
|
+
|
|
55
|
+
```sh
|
|
56
|
+
dsh --profile autoctx --patch "$DSH_HOME/profiles/autoctx/node_modules/dsh-autoctx/off.patch.yml"
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
升级到最新版,或固定到本版本:
|
|
60
|
+
|
|
61
|
+
```sh
|
|
62
|
+
dsh plugin --profile autoctx add dsh-autoctx@latest
|
|
63
|
+
# 或固定版本
|
|
64
|
+
dsh plugin --profile autoctx add dsh-autoctx@1.2.5
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
卸载前,先删除自己在 profile 的 `cordis.patch.yml` 中添加的 `- id: dsh-autoctx` 覆盖项,再执行:
|
|
68
|
+
|
|
69
|
+
```sh
|
|
70
|
+
dsh plugin --profile autoctx remove dsh-autoctx
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
移除 bundle 层会恢复原生压缩。
|
|
74
|
+
|
|
75
|
+
## 设置与默认值
|
|
76
|
+
|
|
77
|
+
Web 界面:左下角 **Settings → DSH AutoCTX**,或 **Plugins → dsh-autoctx → Configure**。可设置模式、学习、上下文选择、自适应思考强度、Jev endpoint、模型路由和 ceiling ratio。输入框徽章显示上下文压力与决策;详情卡显示调度、思考策略及学习账本,manual 模式下还显示待审批提案。`apiKey` 是 secret 字段,不写入日志或记录文件。
|
|
78
|
+
|
|
79
|
+
| 配置键 | 默认值与含义 |
|
|
80
|
+
| --- | --- |
|
|
81
|
+
| `mode` | `auto`;可选 `auto`、`single`、`native` |
|
|
82
|
+
| `apiKey` | 未设置;未设置时使用原生模式 |
|
|
83
|
+
| `jevModel` | `jev-1.13`,计费;`jev-1.13-free` 可用但有限流 |
|
|
84
|
+
| `jevEndpoint` | `https://opencode.ai/inference/v1/systemone` |
|
|
85
|
+
| `modelRoutes` | `{}`;模型 id → dsh provider 名称 |
|
|
86
|
+
| `allowPaidModels` | `false`;是否允许付费路由 |
|
|
87
|
+
| `reviewerModel` | `same`;使用执行模型 fork 审查;`auto` 由 Jev 选择,也可指定模型 id |
|
|
88
|
+
| `reviewBeforeFinish` | `true`;修改文件的根回合结束前启动一个独立审查子代理 |
|
|
89
|
+
| `reviewMaxRequests` | `6`;审查请求上限 |
|
|
90
|
+
| `advisorModel` | `off`;可选 `off`、`auto` 或模型 id |
|
|
91
|
+
| `advisorMaxConsults` | `1`;顾问咨询上限 |
|
|
92
|
+
| `advisorStallSteps` | `6`;停滞步数阈值 |
|
|
93
|
+
| `adaptiveEffort` | `high`;Jev 思考策略要求时使用的 effort;`off` 保留默认 effort |
|
|
94
|
+
| `effortSwitchFreeTokens` | `4096`;决策点切换思考强度时,预计未缓存重读量不超过此值则不询问 Jev |
|
|
95
|
+
| `maxEffortDecisions` | `6`;每个根任务独立的 Jev 思考决策调用上限;不占用 `maxDecisions` |
|
|
96
|
+
| `effortPhaseRequests` | `4`;思考阶段内高强度请求上限;新触发会续期 |
|
|
97
|
+
| `cacheTtlSeconds` | `300`;运行时估算假设的缓存复用寿命;提供方 TTL 未验证 |
|
|
98
|
+
| `learning` | `frozen`;可选 `auto`、`manual`、`frozen` |
|
|
99
|
+
| `contextSelection` | `jev`;可选 `jev`、`summary` |
|
|
100
|
+
| `maxDecisions` | `12`;每个根任务的 Jev 路由/上下文决策调用上限 |
|
|
101
|
+
| `maxChildren` | `2`;子代理数量上限 |
|
|
102
|
+
| `maxConcurrent` | `2`;并发子代理数量上限 |
|
|
103
|
+
| `maxChildRequests` | `16`;子代理请求上限 |
|
|
104
|
+
| `maxChildTurnRequests` | `8`;子代理每回合请求上限 |
|
|
105
|
+
| `pressureTokens` | `64000` tokens;上下文压力阈值 |
|
|
106
|
+
| `ceilingRatio` | `1.5`;达到 `pressureTokens × ceilingRatio` 时必须缩减上下文 |
|
|
107
|
+
| `retainTokens` | `8000` tokens;保留 token 配置 |
|
|
108
|
+
|
|
109
|
+
## 工作方式
|
|
110
|
+
|
|
111
|
+
**决策。** 普通单代理持续累积一份上下文;原生子代理由执行模型自行选择 spawn/fork,原生压缩总结旧上下文。AutoCTX 在特定节点向 Jev 请求结构化决策。Jev 是评分/选择模型,不编写计划或代码。它判断拟议委派是否值得创建子代理及采用哪条路由;在回合开始给出任务标签与思考策略;在上下文压力下选择证据并判断现在还是稍后替换上下文,权衡重新读取未缓存内容的成本与后续请求缩小的收益;当角色设为 `auto` 时选择 reviewer/advisor 模型。
|
|
112
|
+
|
|
113
|
+
思考策略为 `off`、`decision_points` 或 `always`;决策点包括回合开始、测试失败、审查/顾问报告和停滞。1.2.5 为每次触发提供失败测试输出摘录、diff hunk、审查/顾问报告文本或任务开头等证据,并按目录价给出各选项的估算成本:`skip` 留在当前缓存空间;`pulse` 仅一次高强度请求;`phase` 保持高强度,直到触发问题解决(测试通过、报告已处理、取得进展)或达到 `effortPhaseRequests` 次请求。预计未缓存重读量不超过 `effortSwitchFreeTokens` 的低成本切换,以及紧接上下文替换后的切换,不询问 Jev。思考决策使用独立的 `maxEffortDecisions`,不占用 `maxDecisions` 的 12 次路由/上下文决策。缓存复用按实际请求的精确前缀估算,不为缓存改写、重排或丢弃上下文;这些运行时估算均为理论值,非实际账单。
|
|
114
|
+
|
|
115
|
+
临时 Jev 网关/网络失败会重试一次,随后暂停 Jev 60 秒;认证、计费或 HTTP 429 会停止该任务的 Jev 调用,由本地回退接管:原生压缩、不委派。
|
|
116
|
+
|
|
117
|
+
**上下文选择。** `contextSelection: jev` 在上下文超过 `pressureTokens` 时,把旧区域拆成测试失败段、diff hunk、按函数/类划分的代码和段落等证据片段,而非整块输出。Jev 根据明确的待解决问题(例如仍失败的测试)以及必留内容,评估每片证据额外提供了什么。插件本地去重只保留一份;保留的报告会带入其引用的定义;增益低的片段各归档为一行。达到 `pressureTokens × ceilingRatio` 时必须缩减,必要时使用原生压缩。`summary` 则由执行模型总结。
|
|
118
|
+
|
|
119
|
+
**Reviewer。** 修改过文件的根回合结束前,一个审查子代理检查实际改动。简报先包含工作区 git diff 和每个改动 hunk 周围的代码,再附问题与证据账本。`DEFECTS` 报告返回执行模型处理,`LGTM` 结束回合。
|
|
120
|
+
|
|
121
|
+
**Advisor。** 默认关闭。在同一测试上反复修改并重跑仍失败,或出现停滞时,不带工具的顾问模型接收具体问题、原始失败摘录、diff、周边代码和此前尝试。
|
|
122
|
+
|
|
123
|
+
**学习。** 账本按工作区路径区分,存于 dsh storage 的 `dsh_auto`。每个根回合完成后,Jev 对每个使用过的(模型,角色),先判断其具体主张能否用它指明的后续修改/测试运行核验,再评价质量。整体任务成功或执行模型未采纳建议都不是证据;无法评估时,权重和样本数均不变。原生回答的赞/踩也是证据。
|
|
124
|
+
|
|
125
|
+
- `frozen`(默认):仅记录证据,权重不变。
|
|
126
|
+
- `auto`(BETA,默认关闭):自动应用有界更新。
|
|
127
|
+
- `manual`:将提案汇集到一张审批卡片。
|
|
128
|
+
|
|
129
|
+
自动学习有望随时间在保持质量的同时降低成本,但这只是预期,尚未展示;本次基准中 auto 未超过 frozen。
|
|
130
|
+
|
|
131
|
+
## 额外调用与费用
|
|
132
|
+
|
|
133
|
+
Jev 调用、reviewer/advisor 子代理及其执行模型调用、回合结束时的学习调用,均由用户自己的 API 账号承担费用,不是免费服务。`models.json` 模型表中 `jev-1.13` 的目录价为 **每 100 万输入 tokens $0.042,输出 tokens $0**;`jev-1.13-free` 有限流。下方基准费用均为按目录价的理论估算,非实际账单(定义见基准一节)。1.2.5 在 72 个案例中调用 Jev 368 次,1.2.3 frozen 为 233 次;这些调用也占用任务请求上限。
|
|
134
|
+
|
|
135
|
+
## 基准结果与限制
|
|
136
|
+
|
|
137
|
+
仅报告本次运行,不代表普遍优势。每组使用相同的 **72 个评分案例 = 12 个固定 SWE-bench Verified pytest 任务(`bench/tasks.json`)× 3 个执行模型 × 每模型 2 次运行**。执行模型为 `deepseek-v4.1-flash`、`glm-5.3-flash`、`deepseek-v4-flash`。官方 SWE-bench harness **v4.1.0** 判定 resolved;运行完成或 HTTP 200 均不等于通过。
|
|
138
|
+
|
|
139
|
+
共同限制为:**每任务 600 秒、每任务 64 个模型请求(包含 Jev 调用)、案例并行数 3、最多 2 个子代理/2 个并发子代理**。AutoCTX 与 H 的每案例花费上限为 $0.10,B 与 C 使用 harness 默认的 $0.50;各组均无案例的理论费用(目录价)超过 $0.066,因此花费上限未起约束作用。
|
|
140
|
+
|
|
141
|
+
| 组别 | 配置 |
|
|
142
|
+
| --- | --- |
|
|
143
|
+
| B | 单代理;无子代理工具,原生压缩 |
|
|
144
|
+
| C | 原生 spawn/fork,仅执行模型;原生压缩,无 reviewer/advisor/额外模型 |
|
|
145
|
+
| H | 原生异构子代理;在执行模型及额外 3 个模型的每个(模型,effort)组合上提供一个原生子代理工具,由执行模型选择 |
|
|
146
|
+
| AutoCTX 1.2.5 frozen | 新版本;reviewer `auto`、advisor `auto`、相同的额外 3 个模型、`jev-1.13`;frozen 学习 |
|
|
147
|
+
| AutoCTX 1.2.3 frozen / auto | 上一版本,相同 harness;reviewer `auto`、advisor `auto`、相同的额外 3 个模型、`jev-1.13`;分别使用 frozen 和 auto 学习;auto 在同一次运行的案例间沿用账本 |
|
|
148
|
+
|
|
149
|
+
额外 3 个模型为 `glm-5.3-flash`(执行模型为它时改用 `deepseek-v4.1-flash`)、`deepseek-v4-pro`、`claude-haiku-5-5`。1.2.5 与 1.2.3/B/C/H 均运行于 dsh `0.2.0-rc.2`。B/C/H 不加载插件,均早于 1.2.3 运行。B/C 与 AutoCTX 使用同一 harness 入口;H 第 1 轮使用更早的 harness 快照(`run.py` 有差异),任务和上述共同限制相同。
|
|
150
|
+
|
|
151
|
+
**理论费用依据。** 下文所有费用与缓存数据均为**按目录价的理论估算,非实际账单**。提供方报告的缓存命中受路由、账号和驱逐影响,因此各组按实际发送的 prompt 可复用的缓存,以相同假设和价格比较:请求复用同任务、同缓存空间(模型 + thinking/effort 配置[+ tools])内与此前请求共享的最长前缀;任务内不设过期。未缓存输入按目录输入价(列有缓存写入价时用写入价),复用输入按缓存读取价,输出按输出价;Jev 输入 $0.042/百万 tokens,输出免费;包含执行模型、子代理及 Jev 的全部费用。此基准的任务内不设过期假设与运行时 `cacheTtlSeconds` 的寿命假设不同。
|
|
152
|
+
|
|
153
|
+
两种测量方法:`exact` 使用 bench observer 记录的逐消息前缀哈希(AutoCTX 1.2.x);B/C/H 运行时尚无前缀哈希,使用 `log` 重建(每个 session 按追加增长,输入量下降标记上下文改写)。在 12 个任务上用 `deepseek-v4.1-flash` 重跑 B/C、记录前缀哈希(24 个案例)验证:`log` 比 `exact` 的费用低 2.4%(B)和 3.1%(C);在 AutoCTX 上约低 11%,因为上下文替换会改动较早的 prompt 内容却不一定缩短输入。因此下表基线理论费用可能低估约 3%,AutoCTX 溢价略被高估,而非低估。
|
|
154
|
+
|
|
155
|
+
下表费用与缓存率为理论估算,非实际账单。
|
|
156
|
+
|
|
157
|
+
| 指标(每组 72 个案例) | AutoCTX 1.2.5 frozen | AutoCTX 1.2.3 frozen | AutoCTX 1.2.3 auto | B | C | H |
|
|
158
|
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
159
|
+
| Resolved / 72 | 53/72 | 52/72 | 49/72 | 39/72 | 44/72 | 42/72 |
|
|
160
|
+
| 1.2.5 frozen 对该组的配对胜/负(精确 McNemar p) | — | 6/5 (p=1.0) | — | 15/1 (p=0.001) | 12/3 (p=0.035) | 13/2 (p=0.007) |
|
|
161
|
+
| 理论总费用 USD(方法) | $1.738 (exact) | $1.810 (exact) | $1.603 (exact) | $0.908 (log) | $0.975 (log) | $1.027 (log) |
|
|
162
|
+
| 其中理论费用:执行模型 / 子代理 / Jev(USD) | 1.383 / 0.273 / 0.082 | 1.451 / 0.284 / 0.076 | 1.321 / 0.214 / 0.068 | 0.805 / 0.102 / 0 | 0.844 / 0.131 / 0 | 0.867 / 0.160 / 0 |
|
|
163
|
+
| 每个 resolved 案例的理论费用 | $0.0328 | $0.0348 | $0.0327 | $0.0233 | $0.0222 | $0.0244 |
|
|
164
|
+
| 执行模型理论缓存率 | 87.0% | 85.4% | 86.5% | 94.8% | 95.1% | 95.1% |
|
|
165
|
+
| 墙钟时间总和(秒) | 17580 | 18060 | 17670 | 17454 | 16520 | 17076 |
|
|
166
|
+
| Jev 调用成功/总数 | 367/368 | 233/233 | 260/260 | — | — | — |
|
|
167
|
+
|
|
168
|
+
若全部组统一用 `log` 方法,理论总费用为 AutoCTX 1.2.5 frozen $1.547,B 为 $0.908、C 为 $0.975、H 为 $1.027。
|
|
169
|
+
|
|
170
|
+
AutoCTX 在此样本中比 B、C、H 解决更多案例,也更贵;按理论费用,1.2.5 frozen 比 C 总费用约高 70–80%,每个 resolved 案例约高 48%。1.2.5 与 1.2.3 frozen 按相同运行通道配对、使用 `exact` 比较:53 对 52 个 resolved(6 胜 / 5 负,无质量差异);理论总费用降低 4.0%($1.738 对 $1.810),执行模型理论缓存率 87.0% 对 85.4%,Jev 理论费用增加 $0.006。
|
|
171
|
+
|
|
172
|
+
1.2.5 在 72 个案例中的思考决策:201 次低成本切换(不调用 Jev)、86 次 skip、51 次 phase、9 次 pulse、11 次紧接上下文替换、11 次在思考预算用尽后回退;19 个案例达到 64 请求上限(1.2.3 为 18 个)。该上限包含 Jev 调用,因此 AutoCTX 在执行请求更少时就会触顶:这 19 个案例中有 10 个未解决,其 640 次请求里 77 次(12%)为 Jev 调用。B/C/H 也分别有 17/20/16 个案例触顶,但全部为执行请求。本基准保持该共同上限;把 Jev 排除在上限之外的重跑尚未进行。
|
|
173
|
+
|
|
174
|
+
| 执行模型(每组、每模型 24 个案例;resolved / 24) | AutoCTX 1.2.5 frozen | AutoCTX 1.2.3 frozen | AutoCTX 1.2.3 auto | B | C | H |
|
|
175
|
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
176
|
+
| deepseek-v4.1-flash | 18/24 | 17/24 | 16/24 | 11/24 | 15/24 | 14/24 |
|
|
177
|
+
| glm-5.3-flash | 17/24 | 18/24 | 18/24 | 17/24 | 18/24 | 18/24 |
|
|
178
|
+
| deepseek-v4-flash | 18/24 | 17/24 | 15/24 | 11/24 | 11/24 | 10/24 |
|
|
179
|
+
|
|
180
|
+
Auto 运行中,49 个已完成回合进入学习:35/49 个被判无法评估,权重不变;14/49 个应用了小幅更新。此处 auto 未超过 frozen。
|
|
181
|
+
|
|
182
|
+
测量范围仅为上述 12 个 pytest 任务和 3 个 flash 执行模型;重复运行之间,每次共 12 个任务的运行,resolved 数约波动 ±2 个。按理论费用,AutoCTX 在此处比基线更贵;auto learning 仍为 beta,默认关闭。宿主还存在限制:回合中途创建的 fork 只继承已完成回合,因此审查简报须自行携带当前回合证据。
|
|
183
|
+
|
|
184
|
+
## 复现
|
|
185
|
+
|
|
186
|
+
在仓库目录中准备 [bench/README.md](https://github.com/ZhiweiWei-NAMI/dsh-autoctx/blob/main/bench/README.md) 所列前置条件,包括 Docker 与 SWE-bench 镜像、模型网关和 `JEV_API_KEY`,然后运行:
|
|
187
|
+
|
|
188
|
+
```sh
|
|
189
|
+
python3 bench/run.py --model deepseek-v4.1-flash --arms A --tasks bench/tasks.json --out <dir> --plugin . --parallel 3 \
|
|
190
|
+
--extra-models glm-5.3-flash,deepseek-v4-pro,claude-haiku-5-5 --reviewer auto --advisor auto --max-usd 0.10 \
|
|
191
|
+
--jev-model jev-1.13 --learning frozen # 或 --learning auto
|
|
192
|
+
python3 bench/run.py --model deepseek-v4.1-flash --arms B,C --tasks bench/tasks.json --out <dir> --plugin . --parallel 3
|
|
193
|
+
python3 bench/run.py --model deepseek-v4.1-flash --arms H --tasks bench/tasks.json --out <dir> --plugin . --parallel 3 \
|
|
194
|
+
--extra-models glm-5.3-flash,deepseek-v4-pro,claude-haiku-5-5 --max-usd 0.10
|
|
195
|
+
python3 bench/grade.py <dir> --parallel 3
|
|
196
|
+
python3 bench/theoretical_cost.py "A=<dir>/*/A" "B=<dir>/*/B" "C=<dir>/*/C"
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
`<dir>` 替换为输出目录。A 是 AutoCTX;frozen 与 auto 使用各自目录。对全部 3 个执行模型分别运行,每个模型做 2 次运行,独立运行使用不同目录。`glm-5.3-flash` 对应的 `--extra-models` 改为 `deepseek-v4.1-flash,deepseek-v4-pro,claude-haiku-5-5`;`deepseek-v4-flash` 使用示例中的额外模型。重跑相同命令会续跑未完成案例。`--max-usd 0.10` 是 bench observer 的每案例 $0.10 花费上限。`bench/theoretical_cost.py` 按上文方法估算费用与缓存,输出均为理论估算,非实际账单。
|
|
200
|
+
|
|
201
|
+
## 从旧版本升级
|
|
202
|
+
|
|
203
|
+
按上面的升级步骤安装 1.2.5,并对照当前配置表检查 profile 覆盖项。不要沿用旧 README 中的默认模型或安装标签;当前默认是计费的 `jev-1.13`,发布标签为 `v1.2.5`。
|