dsh-github-copilot 0.4.0-alpha.115 → 0.4.0-alpha.116
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +11 -2
- package/CHANGELOG.md +8 -0
- package/CONTRIBUTING.md +9 -0
- package/README.md +23 -6
- package/README.zh.md +18 -6
- package/deployment-baseline.json +23 -2
- package/docs/auto-high-cost.md +132 -0
- package/docs/auto-iteration-review.json +22 -0
- package/docs/auto-task-routing.md +8 -6
- package/docs/automatic-model-routing.md +4 -1
- package/docs/evidence-driven-iteration.md +86 -0
- package/lib/client.js +239 -75
- package/lib/client.js.map +1 -1
- package/lib/index.js +247 -17
- package/lib/remote.js +125 -40
- package/lib/types/authorization-controller.d.ts +2 -0
- package/lib/types/auto-allocation-card.d.ts +9 -0
- package/lib/types/auto-allocation-evidence.d.ts +38 -0
- package/lib/types/auto-allocation.d.ts +24 -0
- package/lib/types/auto-model-host.d.ts +2 -0
- package/lib/types/auto-model-routing.d.ts +6 -2
- package/lib/types/auto-task-classifier.d.ts +1 -1
- package/lib/types/config.d.ts +8 -2
- package/lib/types/model-preferences-card.d.ts +1 -1
- package/lib/types/preview-route.d.ts +1 -1
- package/lib/types/remote.d.ts +3 -0
- package/lib/types/turn-selection-card.d.ts +2 -1
- package/lib/types/turn-selection-host.d.ts +21 -0
- package/lib/types/turn-selection-remote.d.ts +2 -0
- package/lib/types/turn-selection.d.ts +15 -0
- package/package.json +5 -1
- package/scripts/iteration-review.mjs +53 -0
package/AGENTS.md
CHANGED
|
@@ -258,6 +258,14 @@ Never say GPT-6/search works merely because settings, typecheck or a package imp
|
|
|
258
258
|
|
|
259
259
|
## Changing capabilities
|
|
260
260
|
|
|
261
|
+
Auto routing, assessment, cost weights, continuity and assistance changes must
|
|
262
|
+
first read and validate `docs/auto-iteration-review.json` using
|
|
263
|
+
`scripts/iteration-review.mjs`. Follow `docs/evidence-driven-iteration.md`:
|
|
264
|
+
missing data stays explicit, PRs record the decision and validation, and the
|
|
265
|
+
post-delivery review has an owner, sample trigger, metric and action. Never
|
|
266
|
+
upload local diagnostics automatically. See `docs/auto-high-cost.md` for the
|
|
267
|
+
orthogonal user-cost policy and separately gated advisor acceptance.
|
|
268
|
+
|
|
261
269
|
Continuation processing is reasoning-only: never recursively traverse or serialize
|
|
262
270
|
ordinary payload content to impose a whole-request limit. Preserve bounded
|
|
263
271
|
top-level item/ciphertext work, yielding cancellation and turn-revocation checks,
|
|
@@ -276,8 +284,9 @@ Completed-turn Usage diagnostics (#259) are read-only plugin conversation data a
|
|
|
276
284
|
Auto task routing (#258) uses current `model_picker_category` facts, never model
|
|
277
285
|
names, effort or context capacity as a quality category. Hard eligibility/input
|
|
278
286
|
fit precede task/preference policy; unknown category and uncertain task evidence
|
|
279
|
-
remain explicit.
|
|
280
|
-
|
|
287
|
+
remain explicit. Within the first fitting category, apply positive cost weights
|
|
288
|
+
(ordinary 1, marked 0.2) and finite continuity (previous model multiplier 1.5),
|
|
289
|
+
never unconditional retention or extreme-only cost gating. Capture actual reasons once
|
|
281
290
|
per turn without durable decision events or changing Usage.
|
|
282
291
|
`autoSemanticAssessment` is default-on by explicit user rollout request (#267),
|
|
283
292
|
only for locally unknown demand; preserve explicit false opt-out. One bounded concrete
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,13 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.4.0-alpha.116 (prepared)
|
|
4
|
+
|
|
5
|
+
- Add immediate, cross-account High cost marking to existing model-preference rows through strict narrow Remotes and public SettingsForms CAS/readback (#333).
|
|
6
|
+
- Keep cost orthogonal to task difficulty and supplier category. Use positive weights 1/0.2 and finite previous-model multiplier 1.5 within the fitting category, preserving exclusion, capacity, fallback and admitted-turn ownership. Auxiliary classifiers separately prefer eligible unmarked Lightweight models.
|
|
7
|
+
- Expose bounded current-Session Host-lifetime allocation observations through native Agent lookup and the existing selection dialog, with candidate opportunities, conditional expected selections, actual selections and local JSON export. No automatic upload, durable history, quality/billing inference or native Usage changes.
|
|
8
|
+
- Require the previous Auto review in task plans, Agent contract and PR evidence; explicitly record missing live data and post-delivery owner/sample triggers (#332). Record controlled advisor requirements without shipping unverified extra calls.
|
|
9
|
+
- No Core changes, automatic installation, restart or live model calls for validation.
|
|
10
|
+
|
|
3
11
|
## 0.4.0-alpha.115 (prepared)
|
|
4
12
|
|
|
5
13
|
- Default automatic segmented recovery on in the explicitly selected compaction recovery engine (#330). Keep fitting summaries native; handle known oversized input or one typed summary context-limit failure inside the same Core transaction.
|
package/CONTRIBUTING.md
CHANGED
|
@@ -44,3 +44,12 @@ Prepare version alignment in the implementation PR. Important runtime or deliver
|
|
|
44
44
|
After the release-ready PR merges, the successful main CI run calls the reusable Release workflow on that exact SHA. It creates or reconciles the annotated version tag and a draft-first Release, rebuilds and verifies the package, writes and verifies `SHA256SUMS`, checks remote asset digests, and only then publishes an immutable Release. Identical reruns reconcile safely; conflicting tags, releases or bytes fail closed. A tag created by `GITHUB_TOKEN` does not trigger a second workflow, so publication occurs in the gated main workflow rather than depending on recursive tag events. Do not hand-upload a replacement or bypass failed CI.
|
|
45
45
|
|
|
46
46
|
Release-bearing work is complete only after reporting the published Release URL and verified version/commit/assets/checksum, or clearly identifying a concrete publication blocker and pending step. A merged PR or a local tarball alone is not release delivery. After publication, independently verify the Release is not a draft, its prerelease flag, exact tag/commit, asset names and downloaded SHA-256. An install command must include the target `--profile`; never overwrite a shipped preset. Compare the installed package to the downloaded asset, then separately validate activation after restart. HTTP 200 alone does not prove the new plugin loaded.
|
|
47
|
+
|
|
48
|
+
## Auto policy evidence
|
|
49
|
+
|
|
50
|
+
Before Auto routing, assessment, high-cost, continuity or assistance changes,
|
|
51
|
+
read and validate `docs/auto-iteration-review.json`. Follow
|
|
52
|
+
[`docs/evidence-driven-iteration.md`](./docs/evidence-driven-iteration.md) and
|
|
53
|
+
record the prior decision, evidence status/denominator, actual validation and
|
|
54
|
+
post-delivery owner/trigger in the PR. Missing observations are not a pass.
|
|
55
|
+
Never upload raw local exports or infer billing/quality from allocation counts.
|
package/README.md
CHANGED
|
@@ -23,10 +23,27 @@ Visible Models and Credits resolve missing names for saved accounts without acti
|
|
|
23
23
|
|
|
24
24
|
Adding an account now keeps the device-authorization link, selectable URL, one-time code, Copy code action and Cancel together while authorization is pending. After GitHub authorizes, identity and model verification are shown as a separate phase. Adding never changes the global default; use the existing explicit **Switch** action when ready. Technical diagnostics are secondary, and account/turn identity fallbacks never expose opaque account IDs or infer historical identity from today's settings.
|
|
25
25
|
|
|
26
|
-
**Source candidate: `0.4.0-alpha.
|
|
26
|
+
**Source candidate: `0.4.0-alpha.116` (unreleased). Supported host: official DSH / Windows Desktop `0.2.0-rc.2`.** Earlier DSH pins are historical evidence, not supported installation targets. Publication, profile installation and the version loaded by a running Host are separate states. The versioned commands below are for this candidate after publication, not evidence that its assets exist.
|
|
27
27
|
|
|
28
28
|
## What you can do
|
|
29
29
|
|
|
30
|
+
**High-cost Auto:** in **Manage → Model preferences**, mark costly models with
|
|
31
|
+
**High cost**. Cost and task difficulty are independent: within the first
|
|
32
|
+
fitting category, ordinary models have weight `1`, marked models `0.2`, and the
|
|
33
|
+
previous model gets a finite `1.5×` bonus. All candidates keep a nonzero chance;
|
|
34
|
+
marking does not change fixed choices or admitted turns. This is a declared
|
|
35
|
+
policy, not a price/quality ranking or proven savings.
|
|
36
|
+
|
|
37
|
+
The existing turn selection dialog exposes **Auto allocation observations**:
|
|
38
|
+
local retained opportunities, expected shares and actual selections for this
|
|
39
|
+
Session, with a read-only JSON export. No automatic upload or durable history
|
|
40
|
+
is added; restart/eviction loses evidence. `autoAllocationEvidence: false`
|
|
41
|
+
stops future candidate capture without changing routing; previously retained
|
|
42
|
+
observations remain until disposal/eviction/restart. The auxiliary classifier
|
|
43
|
+
prefers unmarked eligible Lightweight models separately. Advisor calls are not
|
|
44
|
+
implemented. See [requirements](./docs/auto-high-cost.md) and the
|
|
45
|
+
[required iteration review](./docs/evidence-driven-iteration.md).
|
|
46
|
+
|
|
30
47
|
| Task | Where to start |
|
|
31
48
|
|---|---|
|
|
32
49
|
| Sign in and manage account models | **Settings → Models → GitHub Copilot → Sign in** |
|
|
@@ -63,10 +80,10 @@ Supply any launcher patches with repeated `--patch /absolute/file`. Require `sup
|
|
|
63
80
|
For a **standalone named profile**:
|
|
64
81
|
|
|
65
82
|
```sh
|
|
66
|
-
dsh plugin --profile web add https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.
|
|
83
|
+
dsh plugin --profile web add https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.116/dsh-github-copilot-0.4.0-alpha.116.tgz
|
|
67
84
|
```
|
|
68
85
|
|
|
69
|
-
For **Desktop**, its native package manager accepts `dsh-github-copilot@0.4.0-alpha.
|
|
86
|
+
For **Desktop**, its native package manager accepts `dsh-github-copilot@0.4.0-alpha.116` after publication. Official rc.2's **Desktop-bundled CLI** also supports reserved-profile plugin management; a global/generic `dsh` shim is not equivalent. Qualify the installed entry before use. See [Desktop CLI qualification](./docs/npm-distribution.md#desktop-bundled-cli-on-official-rc2) and the distinct [standalone offline procedure](./docs/npm-distribution.md#controlled-offline-cli-maintenance-for-standalone-profiles). Neither path authorizes registry-policy bypass, peer patches or configuration deletion.
|
|
70
87
|
|
|
71
88
|
After an approved reload/restart:
|
|
72
89
|
|
|
@@ -220,14 +237,14 @@ Commit attribution uses `Assisted-by` with the actual tool, never the model prov
|
|
|
220
237
|
GitHub Releases and npm distribute the same original verified tarball. Pin a version; verify Release SHA-256 or npm `dist.integrity`. Never repack an immutable release or move/reuse its tag.
|
|
221
238
|
|
|
222
239
|
```sh
|
|
223
|
-
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.
|
|
224
|
-
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.
|
|
240
|
+
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.116/dsh-github-copilot-0.4.0-alpha.116.tgz
|
|
241
|
+
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.116/SHA256SUMS
|
|
225
242
|
sha256sum --check SHA256SUMS
|
|
226
243
|
```
|
|
227
244
|
|
|
228
245
|
```powershell
|
|
229
246
|
$expected = (Get-Content .\SHA256SUMS).Split()[0]
|
|
230
|
-
$actual = (Get-FileHash .\dsh-github-copilot-0.4.0-alpha.
|
|
247
|
+
$actual = (Get-FileHash .\dsh-github-copilot-0.4.0-alpha.116.tgz -Algorithm SHA256).Hash.ToLowerInvariant()
|
|
231
248
|
if ($actual -cne $expected) { throw 'Release checksum mismatch' }
|
|
232
249
|
```
|
|
233
250
|
|
package/README.zh.md
CHANGED
|
@@ -21,10 +21,22 @@ Models 和 Chat 共用不透明、随主题适配的账号下拉层;原生菜
|
|
|
21
21
|
|
|
22
22
|
添加账号时,设备授权链接、可选择的网址、一次性授权码、复制授权码和取消操作会同时显示。GitHub 授权完成后,会单独显示身份与可用模型验证阶段。添加账号不会更改全局默认;准备好后仍需显式执行现有的**切换**操作。技术诊断收在次级折叠区;账号/历史 turn 缺少身份时不会暴露不透明账号 ID,也不会根据当前设置推断历史身份。
|
|
23
23
|
|
|
24
|
-
**源码候选版本:`0.4.0-alpha.
|
|
24
|
+
**源码候选版本:`0.4.0-alpha.116`(尚未发布)。支持宿主:官方 DSH / Windows Desktop `0.2.0-rc.2`。** 较早 DSH pin 仅是历史证据,不是当前安装目标。已发布、已安装到 profile、已被运行中的 Host 加载,是三个不同状态。下方带版本的命令适用于候选版本发布后,不代表对应资产已经存在。
|
|
25
25
|
|
|
26
26
|
## 你可以做什么
|
|
27
27
|
|
|
28
|
+
**高成本 Auto:**在 **Manage → Model preferences** 中用 **High cost** 标记
|
|
29
|
+
模型。成本与任务难度正交:第一个可容纳输入的同类候选池内,普通模型权重为 `1`,
|
|
30
|
+
高成本模型为 `0.2`,上一轮模型仅获得 `1.5×` 的有限加权。每个合格候选都有非零机会;
|
|
31
|
+
标记不改变手动选择或已接纳的 turn。这是路由政策,不是价格/质量排名,也不证明节省费用。
|
|
32
|
+
|
|
33
|
+
现有 turn 选择说明内提供 **Auto allocation observations**,显示当前 Session
|
|
34
|
+
保留的候选机会、预期份额与实际选择,并提供只读 JSON 导出。不自动上传,也不写入持久
|
|
35
|
+
历史;重启或淘汰会丢失观察。设置 `autoAllocationEvidence: false` 可停止后续候选
|
|
36
|
+
记录,不改变路由;此前观察保留至销毁、淘汰或重启。辅助分类器独立优先使用未标记的
|
|
37
|
+
合格 Lightweight 模型。顾问求助尚未实现。参见[完整需求](./docs/auto-high-cost.md)
|
|
38
|
+
与[强制迭代评审流程](./docs/evidence-driven-iteration.md)。
|
|
39
|
+
|
|
28
40
|
| 任务 | 入口 |
|
|
29
41
|
|---|---|
|
|
30
42
|
| 登录并管理账号模型 | **设置 → 模型 → GitHub Copilot → Sign in** |
|
|
@@ -61,10 +73,10 @@ node package/scripts/check-search-composition.mjs --profile-dir /absolute/profil
|
|
|
61
73
|
**独立具名 profile**:
|
|
62
74
|
|
|
63
75
|
```sh
|
|
64
|
-
dsh plugin --profile web add https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.
|
|
76
|
+
dsh plugin --profile web add https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.116/dsh-github-copilot-0.4.0-alpha.116.tgz
|
|
65
77
|
```
|
|
66
78
|
|
|
67
|
-
**Desktop** 原生包管理器在发布后接受 `dsh-github-copilot@0.4.0-alpha.
|
|
79
|
+
**Desktop** 原生包管理器在发布后接受 `dsh-github-copilot@0.4.0-alpha.116`。官方 rc.2 **Desktop 随附的专属 CLI**也支持管理保留 profile;全局/普通 `dsh` shim 不等价。使用前核验实际安装入口,参见[Desktop CLI 核验](./docs/npm-distribution.md#desktop-bundled-cli-on-official-rc2)及独立的[具名 profile 离线流程](./docs/npm-distribution.md#controlled-offline-cli-maintenance-for-standalone-profiles)。两种路径都不授权绕过包源政策、修改 peer 或删除配置。
|
|
68
80
|
|
|
69
81
|
经批准 reload/restart 后:
|
|
70
82
|
|
|
@@ -218,14 +230,14 @@ Agent 从 `node scripts/agent.mjs describe --json`、`doctor --json`及`plan <ta
|
|
|
218
230
|
GitHub Releases 和 npm 分发同一原始已校验 tarball。固定版本并核验 Release SHA-256 或 npm `dist.integrity`;不重打包不可变 Release、不移动/复用 tag。
|
|
219
231
|
|
|
220
232
|
```sh
|
|
221
|
-
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.
|
|
222
|
-
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.
|
|
233
|
+
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.116/dsh-github-copilot-0.4.0-alpha.116.tgz
|
|
234
|
+
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.0-alpha.116/SHA256SUMS
|
|
223
235
|
sha256sum --check SHA256SUMS
|
|
224
236
|
```
|
|
225
237
|
|
|
226
238
|
```powershell
|
|
227
239
|
$expected = (Get-Content .\SHA256SUMS).Split()[0]
|
|
228
|
-
$actual = (Get-FileHash .\dsh-github-copilot-0.4.0-alpha.
|
|
240
|
+
$actual = (Get-FileHash .\dsh-github-copilot-0.4.0-alpha.116.tgz -Algorithm SHA256).Hash.ToLowerInvariant()
|
|
229
241
|
if ($actual -cne $expected) { throw 'Release checksum mismatch' }
|
|
230
242
|
```
|
|
231
243
|
|
package/deployment-baseline.json
CHANGED
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
},
|
|
9
9
|
"package": {
|
|
10
10
|
"name": "dsh-github-copilot",
|
|
11
|
-
"version": "0.4.0-alpha.
|
|
11
|
+
"version": "0.4.0-alpha.116"
|
|
12
12
|
},
|
|
13
13
|
"evidence": {
|
|
14
14
|
"kind": "source-and-synthetic-test-inventory",
|
|
@@ -1228,6 +1228,27 @@
|
|
|
1228
1228
|
{"file": "tests/client-no-model-roles.spec.ts", "name": "does not resurrect a role entry when a retained legacy Remote arrives later"}
|
|
1229
1229
|
]
|
|
1230
1230
|
},
|
|
1231
|
+
{
|
|
1232
|
+
"id": "managed-copilot-high-cost-allocation",
|
|
1233
|
+
"required": true,
|
|
1234
|
+
"sourceMarkers": [
|
|
1235
|
+
{"file": "src/auto-allocation.ts", "marker": "high-cost-v1"},
|
|
1236
|
+
{"file": "src/config.ts", "marker": "highCostModelIds"},
|
|
1237
|
+
{"file": "src/model-preferences-card.ts", "marker": "High cost"},
|
|
1238
|
+
{"file": "src/turn-selection-host.ts", "marker": "allocationSummary"},
|
|
1239
|
+
{"file": "src/auto-allocation-evidence.ts", "marker": "completeHistory: z.literal(false)"},
|
|
1240
|
+
{"file": "scripts/agent.mjs", "marker": "iterationReview"},
|
|
1241
|
+
{"file": "docs/auto-high-cost.md", "marker": "separate acceptance"}
|
|
1242
|
+
],
|
|
1243
|
+
"tests": [
|
|
1244
|
+
{"file": "tests/auto-allocation.spec.ts", "name": "gives every candidate opportunity without an absolute previous-model shortcut"},
|
|
1245
|
+
{"file": "tests/auto-allocation-evidence.spec.ts", "name": "counts each turn once, bounds retention and never recovers missing evidence after removal"},
|
|
1246
|
+
{"file": "tests/auto-model-routing.spec.ts", "name": "keeps costly Fast models eligible and evidence opt-out independent of the selected route"},
|
|
1247
|
+
{"file": "tests/authorization-controller.spec.ts", "name": "saves orthogonal high-cost preferences narrowly without credential or Session reads"},
|
|
1248
|
+
{"file": "tests/client.spec.ts", "name": "saves a high-cost row through its narrow Remote without discovery and retains exclusion state"},
|
|
1249
|
+
{"file": "tests/fixtures/copilot-accounts-persistence-core.fixture.ts", "name": "persists independent native credentials and SettingsForms CAS across profile restart without cross-account OAuth writes"}
|
|
1250
|
+
]
|
|
1251
|
+
},
|
|
1231
1252
|
{
|
|
1232
1253
|
"id": "managed-copilot-auto-intent-continuity",
|
|
1233
1254
|
"required": true,
|
|
@@ -1282,7 +1303,7 @@
|
|
|
1282
1303
|
{"file": "tests/scripts/parent-model-follow.test.mjs", "name": "explicit same-model and different-model child selections both win"},
|
|
1283
1304
|
{"file": "tests/auto-model-routing.spec.ts", "name": "retains legacy length evidence but does not use it as task or supplier category evidence"},
|
|
1284
1305
|
{"file": "tests/auto-model-routing.spec.ts", "name": "uses the task/category policy with one hard-eligible pool for all preferences"},
|
|
1285
|
-
{"file": "tests/auto-model-routing.spec.ts", "name": "
|
|
1306
|
+
{"file": "tests/auto-model-routing.spec.ts", "name": "uses finite continuity but never lets it override task category or input fit"},
|
|
1286
1307
|
{"file": "tests/auto-model-routing.spec.ts", "name": "discloses category fallbacks and never guesses missing category from ID or capacity"},
|
|
1287
1308
|
{"file": "tests/auto-task-assessment.spec.ts", "name": "bounds context, never reads attachment/reasoning bodies, and does not downshift omitted context"},
|
|
1288
1309
|
{"file": "tests/auto-task-assessment.spec.ts", "name": "inherits difficult-task evidence only through explicit bounded continuations"},
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
# High-cost Auto: requirements and evidence
|
|
2
|
+
|
|
3
|
+
Tracking: #333. Evidence-review workflow: #332.
|
|
4
|
+
Post-delivery review: [#334](https://github.com/cloga/dsh-github-copilot/issues/334).
|
|
5
|
+
Advisor qualification: [#335](https://github.com/cloga/dsh-github-copilot/issues/335).
|
|
6
|
+
|
|
7
|
+
## Product contract
|
|
8
|
+
|
|
9
|
+
High cost is a user-owned exact-model-ID preference, not a supplier category,
|
|
10
|
+
quality rating, price quote or difficulty level. The setting is shared across
|
|
11
|
+
accounts, like exclusions. It applies to Lightweight, Versatile and Powerful
|
|
12
|
+
models, including newly advertised IDs without implementation changes.
|
|
13
|
+
Unmarked new models have ordinary weight. Unavailable saved markings survive
|
|
14
|
+
and apply when the ID returns. Exclusion and high cost are independent.
|
|
15
|
+
|
|
16
|
+
The existing Model preferences row gains a High cost checkbox. Changes save
|
|
17
|
+
immediately through a narrow public SettingsForms CAS and confirmed readback;
|
|
18
|
+
they never select a model, discover models, enumerate Sessions or change the
|
|
19
|
+
global default. Unknown preference state is read-only, not an empty marking
|
|
20
|
+
list. Fixed selection is unaffected. An admitted Auto turn retains its captured
|
|
21
|
+
decision across steps, retries and subsequent preference changes.
|
|
22
|
+
|
|
23
|
+
## Routing policy v1
|
|
24
|
+
|
|
25
|
+
1. Account ownership, metadata validity, exclusion, input modality and input
|
|
26
|
+
headroom are hard requirements.
|
|
27
|
+
2. Assess each new turn using existing contextual local/semantic evidence.
|
|
28
|
+
Preserve existing demand categories and supplier-category fallback.
|
|
29
|
+
3. Inside the first fitting category only, give ordinary candidates weight 1
|
|
30
|
+
and marked candidates weight 0.2. Multiply an eligible previous model's
|
|
31
|
+
weight by 1.5. Continuity is finite, never an early unconditional return.
|
|
32
|
+
4. Select from cumulative positive weights using a stable Agent/turn seed.
|
|
33
|
+
Capture the selected ID, candidate weights and conditional expected shares.
|
|
34
|
+
No additional model calls are made to give candidates a chance.
|
|
35
|
+
5. When no model fits, retain the largest-input-budget native recovery path.
|
|
36
|
+
Cost cannot justify choosing a model that cannot fit.
|
|
37
|
+
|
|
38
|
+
These initial parameters are a routing policy to evaluate, not calibrated
|
|
39
|
+
quality or billing facts. A marked previous model has weight 0.3, lower than an
|
|
40
|
+
ordinary peer's 1. An ordinary previous model has 1.5. A lone fitting marked
|
|
41
|
+
model is selected; an all-marked pool remains usable. A marked Fast model can
|
|
42
|
+
answer a simple task: it has a smaller share, not an extreme-only gate.
|
|
43
|
+
Every eligible final-pool candidate has positive opportunity; finite workloads
|
|
44
|
+
need not visit every model. Stable seeds do not prove independent random trials.
|
|
45
|
+
|
|
46
|
+
The auxiliary semantic classifier is a separate call, not another main-turn
|
|
47
|
+
allocation. It prefers an unmarked eligible Lightweight classifier; if only
|
|
48
|
+
marked classifiers qualify, the existing bounded call remains available.
|
|
49
|
+
Within that cost tier it retains advertised-off preference and stable ID order.
|
|
50
|
+
Its model selection must disclose its own policy and charge limits;
|
|
51
|
+
never count it as an ordinary Auto decision or hide auxiliary costs in Chat
|
|
52
|
+
Usage.
|
|
53
|
+
|
|
54
|
+
## Measurements and acceptance
|
|
55
|
+
|
|
56
|
+
Review the previous policy report before changing weights or continuity.
|
|
57
|
+
Synthetic acceptance covers ordinary/marked/all-marked/single pools, every
|
|
58
|
+
supplier category, exclusions, capacity, category changes, stable seeds,
|
|
59
|
+
preference changes and admitted-turn freezing. Distribution checks use large
|
|
60
|
+
declared seed sets and compare actual allocation with conditional expected
|
|
61
|
+
shares, including a marked previous model.
|
|
62
|
+
|
|
63
|
+
Runtime evidence must distinguish a selected decision from native attempted,
|
|
64
|
+
delivered, finished, failed and cancelled calls. Count a decision once per
|
|
65
|
+
Agent/turn, not per tool step. For every final-pool candidate retain its effective
|
|
66
|
+
weight, opportunity and expected share; compare only equivalent policy versions
|
|
67
|
+
and eligibility windows. No-fit decisions have no fitting-pool distribution.
|
|
68
|
+
Report category, assessment source, switching/continuity and marking state at
|
|
69
|
+
decision time, never reconstructed from today's settings.
|
|
70
|
+
|
|
71
|
+
Initial monitoring may be bounded Host-lifetime evidence. State its window,
|
|
72
|
+
retained sample count, eviction and restart limits. Missing history is unknown,
|
|
73
|
+
not zero usage or success. A session-local snapshot is not account-wide traffic.
|
|
74
|
+
Persistent monitoring requires a separately reviewed public storage seam,
|
|
75
|
+
retention/clearing and explicit scope; do not write custom Core history events.
|
|
76
|
+
`autoAllocationEvidence: false` stops future candidate captures without changing
|
|
77
|
+
selection; older retained observations expire through the normal bounded
|
|
78
|
+
store lifecycle. The read-only export performs no automatic file writes/upload.
|
|
79
|
+
|
|
80
|
+
No prompt, answer, tool content, replay, credential, account name or raw Session
|
|
81
|
+
ID belongs in an aggregate review. Keep evidence local; no automatic upload.
|
|
82
|
+
Do not infer credits from token estimates or account-balance deltas. Completion
|
|
83
|
+
is not task success. Any quality comparison requires independently defined
|
|
84
|
+
validation or user feedback, with missing outcomes counted explicitly.
|
|
85
|
+
|
|
86
|
+
After delivery, the next Auto-policy issue owner reviews a declared sample
|
|
87
|
+
target: initially at least 200 eligible opportunities per compared model,
|
|
88
|
+
across at least 20 Sessions, with policy/candidate-set changes separated.
|
|
89
|
+
This is an initial review threshold, not statistical significance. Retain,
|
|
90
|
+
change or defer with reasons and revised collection action if data is missing.
|
|
91
|
+
Review before the next weight change even if the target has not been reached.
|
|
92
|
+
|
|
93
|
+
## Controlled assistance: approved direction, separate acceptance
|
|
94
|
+
|
|
95
|
+
A main model may explicitly request one bounded advisor analysis by naming a
|
|
96
|
+
goal, a specific bottleneck, distinct attempted approaches and references to
|
|
97
|
+
their validation results. Host verification must use public current-Agent/turn
|
|
98
|
+
records, not trust a model's claim that it failed twice. Repeated identical
|
|
99
|
+
attempts and network/auth/permission/quota problems do not establish a reasoning
|
|
100
|
+
bottleneck. Verified failed checks do not prove model incapability.
|
|
101
|
+
|
|
102
|
+
Require per-request user confirmation or a separately explicit standing
|
|
103
|
+
assistance authorization with a budget. Confirming the feature direction alone
|
|
104
|
+
does not authorize live paid calls during development. Verify the following
|
|
105
|
+
public seams before shipping: tool registration, evidence lookup/ownership,
|
|
106
|
+
confirmation and safe resumption, frozen-account binding and native adapter
|
|
107
|
+
composition. Unsupported seams require an explicit limitation or narrower
|
|
108
|
+
user-triggered mode; no Core patch, private registry or second wire.
|
|
109
|
+
|
|
110
|
+
Deduplicate by Agent/turn and bottleneck; one request has one tracked outcome.
|
|
111
|
+
Send only reviewed relevant material, with input/output/time limits and the
|
|
112
|
+
parent cancellation signal. Use a concrete admitted model on the frozen account,
|
|
113
|
+
respect exclusions/capabilities, and prohibit advisor tools, recursive Auto,
|
|
114
|
+
recursive assistance, automatic replay and new retry loops. High-cost marking
|
|
115
|
+
does not establish expertise or guarantee a different model is better.
|
|
116
|
+
|
|
117
|
+
Return advice as a tool result to the unchanged main turn. The main model
|
|
118
|
+
validates the recommendation before applying it. A future next-turn upgrade is
|
|
119
|
+
a separate policy, not an in-turn model switch. Log request, verification status,
|
|
120
|
+
authorization, call outcome, adoption and validation separately. Unverified
|
|
121
|
+
evidence stays unverified; advice returned or a normal finish is not success.
|
|
122
|
+
Extra calls and attributable usage must be disclosed independently, never
|
|
123
|
+
fabricated as native Chat Usage or actual credits.
|
|
124
|
+
|
|
125
|
+
## Delivery boundaries
|
|
126
|
+
|
|
127
|
+
The first implementation targets marking, weighted main-turn allocation and
|
|
128
|
+
reviewable local evidence. It does not claim an advisor tool, autonomous
|
|
129
|
+
escalation, persistent telemetry, calibrated quality, actual cost savings or
|
|
130
|
+
live model success until their own acceptance evidence exists.
|
|
131
|
+
Rollback removes markings or restores the prior reviewed allocation policy
|
|
132
|
+
through a normal versioned change; never rewrite selections or histories.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
{
|
|
2
|
+
"schemaVersion": 1,
|
|
3
|
+
"feature": "autorouting",
|
|
4
|
+
"status": "not-collected",
|
|
5
|
+
"policyVersion": "high-cost-v1",
|
|
6
|
+
"observation": {
|
|
7
|
+
"start": null,
|
|
8
|
+
"end": null,
|
|
9
|
+
"sampleCount": 0,
|
|
10
|
+
"limitations": "No live weighted-allocation dataset collected. Initial synthetic tests do not establish real workload quality, billing, cost savings or live Desktop activation."
|
|
11
|
+
},
|
|
12
|
+
"previousReview": "Issue #332 workflow pilot and #333 high-cost requirements; prior category policy used unconditional continuity.",
|
|
13
|
+
"findings": "Unconditional continuity prevents allocation opportunities. Hard-prioritizing unmarked peers nearly excludes high-cost candidates; extreme-only gating wrongly couples cost to difficulty.",
|
|
14
|
+
"decision": "Implement positive main-turn weights ordinary=1, marked=0.2 and finite previous-model multiplier=1.5 within the fitting supplier category. Capture local bounded opportunities and expected shares. Defer advisor calls until public evidence/authorization/account seams pass their own acceptance.",
|
|
15
|
+
"validation": "Focused tests and 19 focused tooling tests passed with a supplemental unchanged pinned resolver; exact commands/counts are in the implementation PR. Native Client/Host gateways and SettingsForms fiber/restart persistence included. Synthetic source-component browser checks passed at widths 1280/390/195 in both themes. Normal pnpm verify blocked by configured mirror HTTP 404 for exact rc.2 dependency. Required full CI/build/tarball/publication results are recorded separately in the implementation PR and Release. This is not live Desktop/model/billing evidence.",
|
|
16
|
+
"followUp": {
|
|
17
|
+
"owner": "Maintainer or Agent assigned to post-delivery review issue #334",
|
|
18
|
+
"trigger": "Before the next policy change, or after at least 200 eligible opportunities per compared model across at least 20 Sessions, keeping policy/candidate-set windows separate",
|
|
19
|
+
"metric": "Opportunity-adjusted actual versus expected selections, marked replacement/retention, within-pool switching, assessment latency/failures; outcomes and costs only when independently attributable",
|
|
20
|
+
"action": "Review local sanitized aggregates, check missing/evicted coverage, and retain/change/defer with reasons. This threshold is not statistical significance. No automatic upload, quality inference from completion, or credit inference from balance deltas."
|
|
21
|
+
}
|
|
22
|
+
}
|
|
@@ -188,12 +188,14 @@ lightweight -> versatile -> powerful. Unclassified candidates are a final
|
|
|
188
188
|
explicit fallback only, not fabricated categories. Unknown/new category values
|
|
189
189
|
remain unclassified with a bounded catalog diagnostic.
|
|
190
190
|
|
|
191
|
-
Within the first
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
191
|
+
Within the first fitting category, use stable positive weighted Session/turn
|
|
192
|
+
allocation: ordinary weight 1, user-marked high-cost weight 0.2, eligible previous
|
|
193
|
+
model multiplier 1.5. Cost is independent of task difficulty and supplier
|
|
194
|
+
category. There is no absolute continuity shortcut or extreme-only gate.
|
|
195
|
+
These are reviewed policy parameters, not measured quality or prices. Once
|
|
196
|
+
admitted, all steps/retries share the decision. No supplier cache savings are
|
|
197
|
+
claimed. See [high-cost requirements](./auto-high-cost.md) and
|
|
198
|
+
[evidence-driven iteration](./evidence-driven-iteration.md).
|
|
197
199
|
|
|
198
200
|
If no model fits, preserve the existing largest-admissible-input recovery
|
|
199
201
|
branch and its native compaction diagnostics. Explain that no candidate fits;
|
|
@@ -58,7 +58,10 @@ remains authoritative after final framing/attachment projection.
|
|
|
58
58
|
Task preference uses supplier `model_picker_category`, not capacity, model
|
|
59
59
|
names or reasoning effort as quality rankings. [Task policy](./auto-task-routing.md)
|
|
60
60
|
owns the demand matrix, default semantic assessment, category fallback,
|
|
61
|
-
continuity and stable same-category allocation.
|
|
61
|
+
finite continuity and stable positive-weight same-category allocation.
|
|
62
|
+
User-marked high cost is orthogonal to demand and category; see
|
|
63
|
+
[high-cost requirements](./auto-high-cost.md). Future policy changes must read
|
|
64
|
+
the [latest review](./auto-iteration-review.json) before implementation.
|
|
62
65
|
|
|
63
66
|
## Images, compaction and search
|
|
64
67
|
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
# Evidence-driven iteration
|
|
2
|
+
|
|
3
|
+
Complex features must retain the link between observations, decisions and
|
|
4
|
+
follow-up evaluation. A dashboard alone does not ensure contributors read the
|
|
5
|
+
evidence. Auto is the first required pilot (#332).
|
|
6
|
+
|
|
7
|
+
## Before implementation
|
|
8
|
+
|
|
9
|
+
Read the last feature review and unresolved follow-up. State whether the
|
|
10
|
+
hypothesis is retained, changed or deferred. Declare policy version, observation
|
|
11
|
+
window, denominator and limitations: `observed`, `not-collected`,
|
|
12
|
+
`insufficient-sample` or `unavailable`. Missing data is not a passing result.
|
|
13
|
+
Define the expected improvement, guardrails and sample sufficiency before
|
|
14
|
+
analysis. Total turns alone are insufficient when candidate opportunities differ.
|
|
15
|
+
|
|
16
|
+
Record alternative validation when live observations are absent. Synthetic
|
|
17
|
+
distribution tests, types and native fixtures do not prove real workload quality,
|
|
18
|
+
billing or live activation. A review is not permission to implement everything
|
|
19
|
+
previously discussed or collect sensitive data.
|
|
20
|
+
|
|
21
|
+
## Required Auto review
|
|
22
|
+
|
|
23
|
+
`docs/auto-iteration-review.json` is the latest repository-safe decision record.
|
|
24
|
+
The Auto task plan starts by validating and reading it before implementation:
|
|
25
|
+
|
|
26
|
+
```sh
|
|
27
|
+
node scripts/iteration-review.mjs --report docs/auto-iteration-review.json
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
The bounded dependency-free command validates structure, not the truth of
|
|
31
|
+
conclusions. It never scans profiles, calls DSH, reads credentials or uploads
|
|
32
|
+
data. Unknown fields are rejected; invalid input is not echoed. Human review is
|
|
33
|
+
still necessary to remove sensitive prose.
|
|
34
|
+
|
|
35
|
+
For routing, assessment, cost weights, continuity or assistance-policy PRs,
|
|
36
|
+
reference the review and its status, policy version, window/sample denominator,
|
|
37
|
+
decision and validation. Name the post-delivery review owner, trigger, metric
|
|
38
|
+
and action. Unrelated changes use N/A. CI checks the report and workflow wiring;
|
|
39
|
+
it cannot prove an author read the data or understood it.
|
|
40
|
+
|
|
41
|
+
## Measurement contract
|
|
42
|
+
|
|
43
|
+
- Count decisions once per Agent/turn. Tool steps and retries are not new
|
|
44
|
+
decisions. Selected, attempted, delivered, finished, failed, cancelled and
|
|
45
|
+
unknown have different denominators; completion is not task success.
|
|
46
|
+
- For each eligible final-pool candidate capture effective weight, opportunity
|
|
47
|
+
and conditional expected share. Compare summed expected versus actual
|
|
48
|
+
selections, not raw percentages that ignore eligibility.
|
|
49
|
+
- Split category/eligibility changes from within-pool allocation and continuity.
|
|
50
|
+
Keep policy versions and candidate-set changes separate. Stable deterministic
|
|
51
|
+
seeds do not establish independent random trials.
|
|
52
|
+
- Record assessment source/outcome and timing only at observable boundaries;
|
|
53
|
+
adapter-first-text is not HTTP-first-byte.
|
|
54
|
+
- Do not infer credits from tokens, balance deltas or incomplete usage.
|
|
55
|
+
Unknown costs stay unavailable. Cache rates do not prove switching losses.
|
|
56
|
+
|
|
57
|
+
The first high-cost implementation retains bounded, local Host-lifetime
|
|
58
|
+
selection observations. At most 64 live Agent owners and 128 turns per owner
|
|
59
|
+
are retained; summary detail is bounded to 512 model/category/marking/continuity
|
|
60
|
+
strata and explicitly reports truncation. The current Session export includes
|
|
61
|
+
its retained observation timestamps. Restart, disposal and eviction lose
|
|
62
|
+
evidence; there is no complete traffic denominator or durable collector.
|
|
63
|
+
See [high-cost requirements](./auto-high-cost.md).
|
|
64
|
+
|
|
65
|
+
## Privacy
|
|
66
|
+
|
|
67
|
+
Do not collect prompt/answer/tool content, replay, grants, account names or raw
|
|
68
|
+
Session IDs in reports. Do not write custom durable Core history events or
|
|
69
|
+
modify native Usage. Raw exports stay outside Git. Local read-only exports are
|
|
70
|
+
not automatic uploads; sharing requires approval for the specific data and
|
|
71
|
+
destination. Persistence requires a reviewed public storage seam, retention
|
|
72
|
+
and clearing, not an implicit extension of this pilot.
|
|
73
|
+
|
|
74
|
+
## Post-delivery evaluation
|
|
75
|
+
|
|
76
|
+
Publication and effectiveness evaluation are separate milestones. Close the
|
|
77
|
+
delivery issue normally, while retaining a linked review task with an owner and
|
|
78
|
+
declared candidate-opportunity sample trigger. Review before the next policy
|
|
79
|
+
change even if that trigger is not met.
|
|
80
|
+
|
|
81
|
+
At review time, check sufficiency first. Record retain/change/defer, expected
|
|
82
|
+
versus observed behavior, regressions and uncertainty. Replace the latest
|
|
83
|
+
review instead of accumulating empty templates. If data remains absent, defer
|
|
84
|
+
explicitly and revise the collection action; never fabricate a favorable result.
|
|
85
|
+
No automatic reminder, paid advisor invocation, upload or self-tuning is
|
|
86
|
+
authorized by this workflow.
|