dsh-github-copilot 0.4.2-alpha.11 → 0.4.2-alpha.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +12 -6
- package/CHANGELOG.md +23 -1
- package/README.md +13 -12
- package/README.zh.md +14 -10
- package/deployment-baseline.json +8 -3
- package/docs/auto-high-cost.md +90 -20
- package/docs/auto-iteration-review.json +6 -6
- package/docs/auto-task-routing.md +18 -12
- package/docs/automatic-model-routing.md +4 -2
- package/lib/client.js +50 -6
- package/lib/client.js.map +1 -1
- package/lib/diagnostics-analysis.js +1 -1
- package/lib/{diagnostics-types-qvtuvM5E.js → diagnostics-types-D5T4fto9.js} +59 -14
- package/lib/index.js +13 -28
- package/lib/manual-compaction-recovery.js +1 -1
- package/lib/remote.js +45 -2
- package/lib/types/auto-allocation-evidence.d.ts +2 -0
- package/lib/types/auto-allocation.d.ts +21 -3
- package/lib/types/auto-model-routing.d.ts +2 -1
- package/lib/types/turn-selection-host.d.ts +1 -1
- package/lib/types/turn-selection.d.ts +15 -0
- package/package.json +1 -1
package/AGENTS.md
CHANGED
|
@@ -368,12 +368,18 @@ Completed-turn Usage diagnostics (#259) are read-only plugin conversation data a
|
|
|
368
368
|
Auto task routing (#258) uses current `model_picker_category` facts, never model
|
|
369
369
|
names, effort or context capacity as a quality category. Hard eligibility/input
|
|
370
370
|
fit precede task/preference policy; unknown category and uncertain task evidence
|
|
371
|
-
remain explicit.
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
0.
|
|
375
|
-
|
|
376
|
-
|
|
371
|
+
remain explicit. High-cost-v4 retains every eligible fitting model across
|
|
372
|
+
categories with positive weight. Task/preference sets the favored target, not
|
|
373
|
+
hard category exclusion. Per-model category factors are target 1, second 0.25,
|
|
374
|
+
third 0.05, unknown 0.02 in the existing order, then multiplied by cost/recency.
|
|
375
|
+
These are uncalibrated trial factors, not fixed category probability quotas;
|
|
376
|
+
candidate populations affect shares. Ordinary weight 1 receives finite
|
|
377
|
+
previous-model multiplier 1.5; marked base 0.2 decays to 0.02 only for the previous
|
|
378
|
+
model. One-turn recency resets after a different model, not a hidden streak
|
|
379
|
+
counter or task-failure inference. Preserve v1/v2/v3 historical codecs/cohorts and
|
|
380
|
+
their original explanation. Fit is not task
|
|
381
|
+
success; non-target draws are intentional opportunity, not absence/failure of
|
|
382
|
+
target candidates. Never infer incapability or add task-failure escalation.
|
|
377
383
|
Capture actual reasons once
|
|
378
384
|
per turn without durable decision events or changing Usage.
|
|
379
385
|
`autoSemanticAssessment` is default-on by explicit user rollout request (#267),
|
package/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,28 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
## 0.4.2-alpha.
|
|
3
|
+
## 0.4.2-alpha.13 (prepared)
|
|
4
|
+
|
|
5
|
+
- Give every hard-eligible fitting model positive opportunity across supplier
|
|
6
|
+
categories. Replace hard category filtering with per-model target/second/third
|
|
7
|
+
factors 1/0.25/0.05 and explicit unknown 0.02, multiplied by existing cost and
|
|
8
|
+
one-turn recency weights. These are uncalibrated trial coefficients.
|
|
9
|
+
- Explain intentional non-target draws separately from target absence, record
|
|
10
|
+
each candidate's actual category, and preserve unchanged v1/v2/v3 evidence.
|
|
11
|
+
Keep exclusions, capability/account proof, input fit, native no-fit recovery,
|
|
12
|
+
frozen turns and auxiliary classifier ownership unchanged.
|
|
13
|
+
|
|
14
|
+
## 0.4.2-alpha.12 (published)
|
|
15
|
+
|
|
16
|
+
- Restore positive opportunity for all fitting same-category candidates instead
|
|
17
|
+
of v2's hard high-cost deferral. Marked base weight remains 0.2; the previous
|
|
18
|
+
marked model decays tenfold to 0.02, with ordinary continuity unchanged.
|
|
19
|
+
No hidden streak counter, permanent penalty, automatic escalation or task
|
|
20
|
+
success inference. Single/all-marked pools and admitted turns stay usable.
|
|
21
|
+
- Explain high-cost-v3 one-turn recency decay and preserve original v1/v2
|
|
22
|
+
evidence, summaries and diagnostic cohorts. Sequential synthetic simulations
|
|
23
|
+
show fewer repeat selections, not measured production quality or savings.
|
|
24
|
+
|
|
25
|
+
## 0.4.2-alpha.11 (published)
|
|
4
26
|
|
|
5
27
|
- Prefer fitting unmarked models inside the selected supplier category and defer
|
|
6
28
|
high-cost peers. Marked-only fallback stays usable without a continuity bonus.
|
package/README.md
CHANGED
|
@@ -8,18 +8,19 @@
|
|
|
8
8
|
|
|
9
9
|
GitHub Copilot account discovery, contextual Auto routing and hosted search for DSH. The plugin reuses DSH's published native adapter, retaining a profile-wide account default with independent Session next-turn choices; it does not patch Core or maintain a second model catalog.
|
|
10
10
|
|
|
11
|
-
**Stable release: [`0.4.1`](https://github.com/cloga/dsh-github-copilot/releases/tag/v0.4.1); prepared prerelease: `0.4.2-alpha.
|
|
11
|
+
**Stable release: [`0.4.1`](https://github.com/cloga/dsh-github-copilot/releases/tag/v0.4.1); prepared prerelease: `0.4.2-alpha.13`.** `0.4.2-alpha.1` was not published: its release preflight failed, and its immutable tag will not be reused. Supported host: official DSH / Windows Desktop `0.2.0-rc.2`. The stable version is also published on npm's `latest` dist-tag; prereleases remain on their channel tags. Stability-channel promotion does not expand the qualified host baseline or prove live account, model, search or Desktop health. Earlier DSH pins are historical evidence, not supported installation targets. Published, installed in a profile and loaded by a running Host are separate states.
|
|
12
12
|
|
|
13
13
|
**Upgrade from 0.4.0:** that first stable release can interrupt concurrent managed requests with `COPILOT_PREVIEW_CREDENTIAL_CHANGED` during normal same-account native OAuth renewal. In 0.4.1, a proven plugin-owned refresh with unchanged account, endpoint and entitlements preserves already-dispatched HTTP streams; subsequent requests use freshly validated auth and metadata. Unknown/external credential notifications, revocation, cancellation and provider rejections remain explicit—not a promise of invisible recovery or automatic retry. See [the refresh boundary](./docs/copilot-accounts.md#native-oauth-refresh-and-dispatched-requests-385).
|
|
14
14
|
|
|
15
15
|
## What you can do
|
|
16
16
|
|
|
17
17
|
**High-cost Auto:** in **Manage → Model preferences**, mark costly models with
|
|
18
|
-
**High cost**. Cost and task difficulty are independent:
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
18
|
+
**High cost**. Cost and task difficulty are independent: across every category,
|
|
19
|
+
all eligible fitting candidates retain a nonzero chance. Ordinary
|
|
20
|
+
weight `1` gets a finite continuity bonus (`1.5×` for the previous model).
|
|
21
|
+
High-cost base weight is `0.2`; if that model was used in the previous turn,
|
|
22
|
+
its weight decays tenfold to `0.02` instead of receiving a bonus.
|
|
23
|
+
This one-turn recency penalty is a declared trial, not calibrated quality.
|
|
23
24
|
Marking does not change fixed choices or admitted turns. This is a declared
|
|
24
25
|
policy, not a price/quality ranking or proven savings.
|
|
25
26
|
|
|
@@ -128,10 +129,10 @@ Supply any launcher patches with repeated `--patch /absolute/file`. Require `sup
|
|
|
128
129
|
For a **standalone named profile**:
|
|
129
130
|
|
|
130
131
|
```sh
|
|
131
|
-
dsh plugin --profile web add https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.
|
|
132
|
+
dsh plugin --profile web add https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.13/dsh-github-copilot-0.4.2-alpha.13.tgz
|
|
132
133
|
```
|
|
133
134
|
|
|
134
|
-
For **Desktop**, its native package manager accepts an exact npm spec, for example `dsh-github-copilot@0.4.2-alpha.
|
|
135
|
+
For **Desktop**, its native package manager accepts an exact npm spec, for example `dsh-github-copilot@0.4.2-alpha.13`; stable versions publish on npm's `latest` dist-tag and prereleases on their channel tags. The official rc.2 **Desktop-bundled CLI** also supports reserved-profile plugin management; a global/generic `dsh` shim is not equivalent. Qualify the installed entry before use. See [Desktop CLI qualification](./docs/npm-distribution.md#desktop-bundled-cli-on-official-rc2) and the distinct [standalone offline procedure](./docs/npm-distribution.md#controlled-offline-cli-maintenance-for-standalone-profiles). Neither path authorizes registry-policy bypass, peer patches or configuration deletion.
|
|
135
136
|
|
|
136
137
|
After an approved reload/restart:
|
|
137
138
|
|
|
@@ -164,7 +165,7 @@ All three Auto preferences use the same account-verified, non-excluded eligible
|
|
|
164
165
|
| Complex | Powerful | Powerful | Powerful |
|
|
165
166
|
| Unknown | Versatile | Versatile | Powerful |
|
|
166
167
|
|
|
167
|
-
Categories come from authenticated supplier metadata, not model names, context capacity or measured quality rankings.
|
|
168
|
+
Categories come from authenticated supplier metadata, not model names, context capacity or measured quality rankings. Every eligible fitting model retains positive opportunity across categories. The table above is the preferred target, not a hard filter: category factors are target `1`, second `0.25`, third `0.05`, unknown `0.02`, multiplied by cost/recency weights. These are uncalibrated trial coefficients, not fixed category probability quotas; candidate counts also affect shares. The penalty applies to the previous high-cost model, not other marked peers, and resets after a different model's turn. Fit does not prove task success; non-target selection is intentional opportunity, not task-failure escalation. One real answer model stays fixed throughout each admitted turn's steps and retries; compaction preserves its independently resolved native summary route.
|
|
168
169
|
|
|
169
170
|
Semantic assessment is **enabled by default only for locally unknown tasks**. It makes at most one bounded auxiliary call with a **30-second default end-to-end deadline** and **128-token output budget**. Set `github-copilot.autoSemanticAssessment: false` to opt out. The native plugin configuration exposes `autoSemanticAssessmentTimeoutMs` (1000–120000 ms; `8000` restores the old budget) and `autoSemanticAssessmentModel` (empty for automatic selection, or an exact current-account model ID). They apply to subsequent assessments, not admitted retries. An unavailable/excluded/unsuitable explicit classifier reports unavailable, never silently switches. Automatic selection is metadata-driven, not measured optimal; the new default is a trial, not proof of past timeout causes. See [assessment settings](./docs/auto-task-routing.md#using-auto).
|
|
170
171
|
|
|
@@ -321,14 +322,14 @@ Commit attribution uses `Assisted-by` with the actual tool, never the model prov
|
|
|
321
322
|
GitHub Releases and npm distribute the same original verified tarball. Pin a version; verify Release SHA-256 or npm `dist.integrity`. Never repack an immutable release or move/reuse its tag.
|
|
322
323
|
|
|
323
324
|
```sh
|
|
324
|
-
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.
|
|
325
|
-
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.
|
|
325
|
+
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.13/dsh-github-copilot-0.4.2-alpha.13.tgz
|
|
326
|
+
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.13/SHA256SUMS
|
|
326
327
|
sha256sum --check SHA256SUMS
|
|
327
328
|
```
|
|
328
329
|
|
|
329
330
|
```powershell
|
|
330
331
|
$expected = (Get-Content .\SHA256SUMS).Split()[0]
|
|
331
|
-
$actual = (Get-FileHash .\dsh-github-copilot-0.4.2-alpha.
|
|
332
|
+
$actual = (Get-FileHash .\dsh-github-copilot-0.4.2-alpha.13.tgz -Algorithm SHA256).Hash.ToLowerInvariant()
|
|
332
333
|
if ($actual -cne $expected) { throw 'Release checksum mismatch' }
|
|
333
334
|
```
|
|
334
335
|
|
package/README.zh.md
CHANGED
|
@@ -8,16 +8,20 @@
|
|
|
8
8
|
|
|
9
9
|
为 DSH 提供 GitHub Copilot 账号模型发现、上下文 Auto 选模和 hosted search。插件复用 DSH 公开的原生适配器,保留 profile 全局默认账号,并支持 Session 后续 turn 指定独立账号;不修改 Core,也不维护第二套模型目录。
|
|
10
10
|
|
|
11
|
-
**稳定版:[`0.4.1`](https://github.com/cloga/dsh-github-copilot/releases/tag/v0.4.1);待发布预览版:`0.4.2-alpha.
|
|
11
|
+
**稳定版:[`0.4.1`](https://github.com/cloga/dsh-github-copilot/releases/tag/v0.4.1);待发布预览版:`0.4.2-alpha.13`。** `0.4.2-alpha.1` 的发布预检失败,因此未发布;其不可变 tag 不会复用。支持宿主:官方 DSH / Windows Desktop `0.2.0-rc.2`。稳定版也发布到 npm 的 `latest` dist-tag;预发布版本仍使用各自渠道 tag。稳定渠道不扩大已验证的宿主基线,也不证明实时账号、模型、搜索或 Desktop 健康状态。较早 DSH pin 仅是历史证据,不是当前安装目标。已发布、已安装到 profile、已被运行中的 Host 加载,是三个不同状态。
|
|
12
12
|
|
|
13
13
|
**从 0.4.0 升级:**首个正式版在同账号正常原生 OAuth 刷新时,可能以 `COPILOT_PREVIEW_CREDENTIAL_CHANGED` 中止并行托管请求。0.4.1 仅在插件拥有的原生刷新被精确验证,且账号、端点与权限连续时,保留已发出的 HTTP 流;后续请求重新验证认证与模型元数据。未知/外部凭据通知、撤销、取消与供应商拒绝仍明确报错,不承诺所有刷新无感,也不自动重试。参见[刷新边界](./docs/copilot-accounts.md#native-oauth-refresh-and-dispatched-requests-385)。
|
|
14
14
|
|
|
15
15
|
## 你可以做什么
|
|
16
16
|
|
|
17
17
|
**高成本 Auto:**在 **Manage → Model preferences** 中用 **High cost** 标记
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
18
|
+
模型。成本与任务难度正交:所有分类中合格且可容纳输入的候选保留非零机会。
|
|
19
|
+
任务分类只影响权重,不再硬筛选分类:目标、第二、第三分类分别乘 `1 / 0.25 / 0.05`,
|
|
20
|
+
分类未知明确使用 `0.02`,再乘成本与近期权重。这些系数是未校准试验,不是固定分类概率配额;
|
|
21
|
+
各分类模型数量也影响份额。非目标分类抽中不表示目标模型不存在或已经失败。
|
|
22
|
+
普通权重 `1` 可获得上一轮 `1.5×` 有限加权;高成本基础权重 `0.2`,
|
|
23
|
+
若上一轮用了该模型,本轮权重十倍衰减到 `0.02`,而不是获得连续性奖励。
|
|
24
|
+
这是一轮近期惩罚的声明试行参数,不是经过校准的质量判断;
|
|
21
25
|
标记不改变手动选择或已接纳的 turn。这是路由政策,不是价格/质量排名,也不证明节省费用。
|
|
22
26
|
|
|
23
27
|
现有 turn 选择说明内提供 **Auto allocation observations**,显示当前 Session
|
|
@@ -107,10 +111,10 @@ node package/scripts/check-search-composition.mjs --profile-dir /absolute/profil
|
|
|
107
111
|
**独立具名 profile**:
|
|
108
112
|
|
|
109
113
|
```sh
|
|
110
|
-
dsh plugin --profile web add https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.
|
|
114
|
+
dsh plugin --profile web add https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.13/dsh-github-copilot-0.4.2-alpha.13.tgz
|
|
111
115
|
```
|
|
112
116
|
|
|
113
|
-
**Desktop** 原生包管理器接受精确 npm spec,例如 `dsh-github-copilot@0.4.2-alpha.
|
|
117
|
+
**Desktop** 原生包管理器接受精确 npm spec,例如 `dsh-github-copilot@0.4.2-alpha.13`;稳定版发布到 npm `latest` dist-tag,预发布版使用对应渠道 tag。官方 rc.2 **Desktop 随附的专属 CLI**也支持管理保留 profile;全局/普通 `dsh` shim 不等价。使用前核验实际安装入口,参见[Desktop CLI 核验](./docs/npm-distribution.md#desktop-bundled-cli-on-official-rc2)及独立的[具名 profile 离线流程](./docs/npm-distribution.md#controlled-offline-cli-maintenance-for-standalone-profiles)。两种路径都不授权绕过包源政策、修改 peer 或删除配置。
|
|
114
118
|
|
|
115
119
|
经批准 reload/restart 后:
|
|
116
120
|
|
|
@@ -143,7 +147,7 @@ dsh plugin --profile web add https://github.com/cloga/dsh-github-copilot/release
|
|
|
143
147
|
| 复杂 | Powerful | Powerful | Powerful |
|
|
144
148
|
| 未知 | Versatile | Versatile | Powerful |
|
|
145
149
|
|
|
146
|
-
|
|
150
|
+
分类来自认证后的供应方元数据,不按模型名、上下文容量或臆测质量排名。上表是偏向的目标分类,不是硬筛选;所有分类中合格且可容纳的模型均按正权重抽选。惩罚只针对上一轮高成本模型,不影响其它已标记模型;换用其它模型一轮后恢复基础权重。可容纳不证明能解决任务,非目标分类抽中不表示目标模型已经失败,不自动升级求助。目标分类实际缺少可容纳候选时明确说明。同一已准入 turn 的工具步骤和重试保持同一真实回答模型;压缩保留独立解析的原生摘要路由。
|
|
147
151
|
|
|
148
152
|
语义判断**默认开启,仅用于本地需求未知的任务**。最多一次辅助请求,**默认 30 秒端到端期限、128 tokens 输出预算**。可用 `github-copilot.autoSemanticAssessment: false` 关闭。原生插件配置提供 `autoSemanticAssessmentTimeoutMs`(1000–120000 毫秒,`8000` 恢复旧预算)和 `autoSemanticAssessmentModel`(空字符串自动选择,或当前账号的精确模型 ID)。只影响后续评估,不改变已准入 turn 的重试。指定分类模型不存在/被排除/不合适时明确报告 unavailable,不偷偷换模型。自动选择依据元数据,不代表实测最优;30 秒是试验默认值,不证明过去主要因超时。参见[评估配置](./docs/auto-task-routing.md#using-auto)。
|
|
149
153
|
|
|
@@ -297,14 +301,14 @@ Agent 从 `node scripts/agent.mjs describe --json`、`doctor --json`及`plan <ta
|
|
|
297
301
|
GitHub Releases 和 npm 分发同一原始已校验 tarball。固定版本并核验 Release SHA-256 或 npm `dist.integrity`;不重打包不可变 Release、不移动/复用 tag。
|
|
298
302
|
|
|
299
303
|
```sh
|
|
300
|
-
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.
|
|
301
|
-
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.
|
|
304
|
+
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.13/dsh-github-copilot-0.4.2-alpha.13.tgz
|
|
305
|
+
curl -LO https://github.com/cloga/dsh-github-copilot/releases/download/v0.4.2-alpha.13/SHA256SUMS
|
|
302
306
|
sha256sum --check SHA256SUMS
|
|
303
307
|
```
|
|
304
308
|
|
|
305
309
|
```powershell
|
|
306
310
|
$expected = (Get-Content .\SHA256SUMS).Split()[0]
|
|
307
|
-
$actual = (Get-FileHash .\dsh-github-copilot-0.4.2-alpha.
|
|
311
|
+
$actual = (Get-FileHash .\dsh-github-copilot-0.4.2-alpha.13.tgz -Algorithm SHA256).Hash.ToLowerInvariant()
|
|
308
312
|
if ($actual -cne $expected) { throw 'Release checksum mismatch' }
|
|
309
313
|
```
|
|
310
314
|
|
package/deployment-baseline.json
CHANGED
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
},
|
|
9
9
|
"package": {
|
|
10
10
|
"name": "dsh-github-copilot",
|
|
11
|
-
"version": "0.4.2-alpha.
|
|
11
|
+
"version": "0.4.2-alpha.13"
|
|
12
12
|
},
|
|
13
13
|
"evidence": {
|
|
14
14
|
"kind": "source-and-synthetic-test-inventory",
|
|
@@ -1342,6 +1342,7 @@
|
|
|
1342
1342
|
"sourceMarkers": [
|
|
1343
1343
|
{"file": "src/auto-allocation.ts", "marker": "high-cost-v1"},
|
|
1344
1344
|
{"file": "src/auto-allocation.ts", "marker": "high-cost-v2"},
|
|
1345
|
+
{"file": "src/auto-allocation.ts", "marker": "high-cost-v4"},
|
|
1345
1346
|
{"file": "src/auto-allocation.ts", "marker": "deferredHighCostCount"},
|
|
1346
1347
|
{"file": "src/config.ts", "marker": "highCostModelIds"},
|
|
1347
1348
|
{"file": "src/model-preferences-card.ts", "marker": "High cost"},
|
|
@@ -1351,7 +1352,9 @@
|
|
|
1351
1352
|
{"file": "docs/auto-high-cost.md", "marker": "separate acceptance"}
|
|
1352
1353
|
],
|
|
1353
1354
|
"tests": [
|
|
1354
|
-
{"file": "tests/auto-allocation.spec.ts", "name": "simulates 20000 turns per previous state
|
|
1355
|
+
{"file": "tests/auto-allocation.spec.ts", "name": "simulates 20000 turns per previous state with positive cost-decayed opportunity"},
|
|
1356
|
+
{"file": "tests/auto-allocation.spec.ts", "name": "decays a marked previous model tenfold without eliminating its opportunity"},
|
|
1357
|
+
{"file": "tests/auto-allocation.spec.ts", "name": "keeps v2 hard-deferral evidence readable without relabeling it as v3"},
|
|
1355
1358
|
{"file": "tests/auto-allocation.spec.ts", "name": "keeps v1 historical weights readable without relabeling them as v2"},
|
|
1356
1359
|
{"file": "tests/auto-allocation-evidence.spec.ts", "name": "keeps legacy-only summaries v1 and separates mixed-policy decisions explicitly"},
|
|
1357
1360
|
{"file": "tests/auto-allocation-evidence.spec.ts", "name": "counts each turn once, bounds retention and never recovers missing evidence after removal"},
|
|
@@ -1417,7 +1420,9 @@
|
|
|
1417
1420
|
{"file": "tests/scripts/parent-model-follow.test.mjs", "name": "explicit same-model and different-model child selections both win"},
|
|
1418
1421
|
{"file": "tests/auto-model-routing.spec.ts", "name": "retains legacy length evidence but does not use it as task or supplier category evidence"},
|
|
1419
1422
|
{"file": "tests/auto-model-routing.spec.ts", "name": "uses the task/category policy with one hard-eligible pool for all preferences"},
|
|
1420
|
-
{"file": "tests/auto-model-routing.spec.ts", "name": "uses finite continuity
|
|
1423
|
+
{"file": "tests/auto-model-routing.spec.ts", "name": "uses finite continuity as a multiplier and never lets it override input fit"},
|
|
1424
|
+
{"file": "tests/auto-category-distribution.spec.ts", "name": "retains positive opportunity for every fitting model for %s demand"},
|
|
1425
|
+
{"file": "tests/auto-category-distribution.spec.ts", "name": "attributes each candidate to its own category in bounded summaries and persistent cohorts"},
|
|
1421
1426
|
{"file": "tests/auto-model-routing.spec.ts", "name": "discloses category fallbacks and never guesses missing category from ID or capacity"},
|
|
1422
1427
|
{"file": "tests/auto-task-assessment.spec.ts", "name": "bounds context, never reads attachment/reasoning bodies, and does not downshift omitted context"},
|
|
1423
1428
|
{"file": "tests/auto-task-assessment.spec.ts", "name": "inherits difficult-task evidence only through explicit bounded continuations"},
|
package/docs/auto-high-cost.md
CHANGED
|
@@ -42,35 +42,61 @@ unchanged official Core fixture, not that the affected live Host read the mark
|
|
|
42
42
|
before the historical Astra decision. Do not hardcode a model, change live
|
|
43
43
|
preferences or rewrite a frozen explanation to manufacture a repair.
|
|
44
44
|
|
|
45
|
-
## Routing policy
|
|
45
|
+
## Routing policy v4 (#414)
|
|
46
46
|
|
|
47
47
|
1. Account ownership, metadata validity, exclusion, input modality and input
|
|
48
48
|
headroom are hard requirements.
|
|
49
49
|
2. Assess each new turn using existing contextual local/semantic evidence.
|
|
50
|
-
Preserve existing demand categories
|
|
51
|
-
3.
|
|
52
|
-
|
|
50
|
+
Preserve existing demand categories as the target, not a hard pool filter.
|
|
51
|
+
3. Retain every eligible fitting candidate across categories. Multiply each
|
|
52
|
+
model's cost/recency weight by its category factor: target 1, second 0.25,
|
|
53
|
+
third 0.05, unknown 0.02. The order is Powerful/Versatile/Lightweight for a
|
|
54
|
+
Powerful target, Lightweight/Versatile/Powerful for Lightweight, and
|
|
55
|
+
Versatile/Powerful/Lightweight for Versatile. Unknown stays explicit, never
|
|
56
|
+
inferred from names or capacities. Missing protocol/capability proof remains
|
|
57
|
+
a hard admission failure, not an unknown-category opportunity.
|
|
53
58
|
Ordinary candidates have weight 1 and an eligible previous ordinary model
|
|
54
|
-
receives multiplier 1.5. Marked candidates have weight 0.2
|
|
59
|
+
receives multiplier 1.5. Marked candidates have base weight 0.2; the previous
|
|
60
|
+
marked model decays tenfold to 0.02 instead of receiving a continuity bonus.
|
|
55
61
|
4. Select from cumulative positive weights using a stable Agent/turn seed.
|
|
56
|
-
Capture the selected ID, candidate
|
|
62
|
+
Capture the selected ID, each candidate's actual category and category factor,
|
|
63
|
+
final weights and conditional expected shares.
|
|
57
64
|
No additional model calls are made to give candidates a chance.
|
|
58
65
|
5. When no model fits, retain the largest-input-budget native recovery path.
|
|
59
66
|
Cost cannot justify choosing a model that cannot fit.
|
|
60
67
|
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
68
|
+
V4 corrects v3's hard category exclusion: a fitting non-target category now has
|
|
69
|
+
positive opportunity, even while the target has fitting models. Category factors
|
|
70
|
+
are deliberately biased but uncalibrated trial coefficients; task-quality,
|
|
71
|
+
switching-cost and real workload acceptance remain unverified. More models in
|
|
72
|
+
a category increase its total opportunity; these are per-model factors, not
|
|
73
|
+
fixed category probability quotas. A category with only high-cost models can
|
|
74
|
+
have lower total probability than an ordinary non-target category.
|
|
75
|
+
|
|
76
|
+
V3 corrected v2's over-strong hard deferral after the user clarified that high
|
|
77
|
+
cost should have rare opportunity and strongly decay after selection, not become
|
|
78
|
+
permanently unavailable. The 10x repeat penalty is a declared trial, not a
|
|
79
|
+
calibrated optimum. It uses existing previous-route evidence only; there is no
|
|
80
|
+
hidden multi-turn counter, persistent penalty, time cooldown or failed-task guess.
|
|
81
|
+
If the same marked model repeats, it remains at 0.02 rather than recursively
|
|
82
|
+
shrinking. A different model's turn resets it to 0.2. Other marked models retain
|
|
83
|
+
base weight. A lone fitting marked model is selected and an all-marked pool
|
|
84
|
+
remains usable. Fit and supplier
|
|
66
85
|
category cannot prove peers can solve the task; no automatic escalation, extra
|
|
67
|
-
calls
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
86
|
+
calls or in-turn switch is introduced. V4 intentionally permits non-target
|
|
87
|
+
categories; it does not infer they can solve the task.
|
|
88
|
+
Every fitting candidate has positive opportunity. In an isolated same-category
|
|
89
|
+
two-model pool, with one ordinary
|
|
90
|
+
peer, a marked model has share 2/17 (11.8%) when the ordinary model was previous,
|
|
91
|
+
then 1/51 (2.0%) when it was itself previous. The independent-draw approximation
|
|
92
|
+
for a predeclared pair is 2/867 (0.23%), not the probability of any streak in a
|
|
93
|
+
long Session or a guarantee against repeats. Stable seeds do not prove
|
|
94
|
+
independent random trials. Conditional shares are not observed billing.
|
|
71
95
|
|
|
72
|
-
Historical v1
|
|
73
|
-
|
|
96
|
+
Historical v1 retains mixed-pool weights and marked continuity; v2 retains hard
|
|
97
|
+
cost-tier deferral and its deferred count; v3 retains hard category filtering
|
|
98
|
+
with positive cost/recency weights. All remain readable unchanged.
|
|
99
|
+
Persistent cohorts retain original policy versions.
|
|
74
100
|
The Session summary separates incompatible policy cohorts and reports omitted
|
|
75
101
|
decisions instead of relabeling them or merging different opportunities.
|
|
76
102
|
|
|
@@ -81,7 +107,9 @@ five Sol turns followed by two Astra turns; its last 178 messages belonged to
|
|
|
81
107
|
one frozen turn. This does not establish a distribution violation. A conditional
|
|
82
108
|
two-peer seeded reproduction matches the two Astra choices, but the exact
|
|
83
109
|
historical pool was not retained. V2 implements the user's stricter preference,
|
|
84
|
-
not a claimed repair to a proven random-number defect.
|
|
110
|
+
not a claimed repair to a proven random-number defect. V3 supersedes v2's hard
|
|
111
|
+
exclusion with positive recency-decayed opportunity, without a live performance
|
|
112
|
+
claim.
|
|
85
113
|
|
|
86
114
|
The auxiliary semantic classifier is a separate call, not another main-turn
|
|
87
115
|
allocation. It prefers an unmarked eligible Lightweight classifier; if only
|
|
@@ -92,14 +120,56 @@ never count it as an ordinary Auto decision or hide auxiliary costs in Chat
|
|
|
92
120
|
Usage.
|
|
93
121
|
|
|
94
122
|
## Measurements and acceptance
|
|
123
|
+
|
|
124
|
+
### Reproducible v4 distribution trial
|
|
125
|
+
|
|
126
|
+
From a source checkout with dependencies installed and Node 24:
|
|
127
|
+
|
|
128
|
+
```sh
|
|
129
|
+
node --experimental-transform-types scripts/simulate-auto-distribution.mjs
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
This calls the real selector for 19 scenarios, 100000 draws each, using one
|
|
133
|
+
synthetic Session and ascending turns. Six synthetic models have two per
|
|
134
|
+
category; only Powerful A is marked. All capacities are 128000 with 8000 output
|
|
135
|
+
unless a scenario explicitly changes fit. IDs are fixtures, not an account
|
|
136
|
+
catalog. No credentials, discovery, model calls or profile mutation occur.
|
|
137
|
+
Rows report counts, averaged conditional expected share and observed frequency;
|
|
138
|
+
every positive-share row must be observed and differ by less than 0.5 percentage
|
|
139
|
+
points. Seed determinism does not establish independent random trials.
|
|
140
|
+
|
|
141
|
+
With no previous model, the twelve task/preference combinations collapse to
|
|
142
|
+
these three distinct pools. Observed percentages for 100000 draws each:
|
|
143
|
+
|
|
144
|
+
| Preferred target | Powerful A marked | Powerful B ordinary | Versatile A | Versatile B | Lightweight A | Lightweight B |
|
|
145
|
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
146
|
+
| Lightweight | 0.355 | 1.985 | 9.793 | 9.619 | 39.001 | 39.247 |
|
|
147
|
+
| Versatile | 2.157 | 10.346 | 41.589 | 41.725 | 2.058 | 2.125 |
|
|
148
|
+
| Powerful | 11.080 | 55.708 | 13.830 | 13.776 | 2.750 | 2.856 |
|
|
149
|
+
|
|
150
|
+
The respective theoretical percentages are:
|
|
151
|
+
Lightweight 0.390625/1.953125/9.765625/9.765625/39.0625/39.0625;
|
|
152
|
+
Versatile 2.083333/10.416667/41.666667/41.666667/2.083333/2.083333;
|
|
153
|
+
Powerful 11.111111/55.555556/13.888889/13.888889/2.777778/2.777778.
|
|
154
|
+
The task/preference target matrix still applies; it biases these probabilities.
|
|
155
|
+
|
|
156
|
+
For a Powerful target, fixed previous ordinary gives the marked model theoretical
|
|
157
|
+
8.6957%, observed 8.663%; fixed previous marked gives 1.2346%, observed 1.296%.
|
|
158
|
+
The sequential simulation updates previous after every draw: marked 8.577%,
|
|
159
|
+
ordinary Powerful 61.215%, Versatile 12.705%/12.632%, Lightweight 2.410%/2.461%,
|
|
160
|
+
with 93 adjacent marked pairs. These are fixture frequencies, not predictions
|
|
161
|
+
for current account pools. The script also covers both Powerful models marked,
|
|
162
|
+
one/both Powerful models failing input fit and mixed image support.
|
|
163
|
+
Hard-ineligible rows retain zero opportunity; there is no guaranteed finite
|
|
164
|
+
waiting time for any model.
|
|
95
165
|
|
|
96
166
|
Review the previous policy report before changing weights or continuity.
|
|
97
167
|
Synthetic acceptance covers ordinary/marked/all-marked/single pools, every
|
|
98
168
|
supplier category, exclusions, capacity, category changes, stable seeds,
|
|
99
169
|
preference changes and admitted-turn freezing. Distribution checks use large
|
|
100
170
|
declared seed sets and compare actual allocation with conditional expected
|
|
101
|
-
shares, including
|
|
102
|
-
|
|
171
|
+
shares, including marked previous models, all-marked/single pools, sequential
|
|
172
|
+
previous-route updates and backward-compatible v1/v2 evidence.
|
|
103
173
|
|
|
104
174
|
Runtime evidence must distinguish a selected decision from native attempted,
|
|
105
175
|
delivered, finished, failed and cancelled calls. Count a decision once per
|
|
@@ -2,21 +2,21 @@
|
|
|
2
2
|
"schemaVersion": 1,
|
|
3
3
|
"feature": "autorouting",
|
|
4
4
|
"status": "insufficient-sample",
|
|
5
|
-
"policyVersion": "high-cost-
|
|
5
|
+
"policyVersion": "high-cost-v4",
|
|
6
6
|
"observation": {
|
|
7
7
|
"start": "2026-10-08T00:00:00Z",
|
|
8
8
|
"end": "2026-10-09T23:59:59Z",
|
|
9
9
|
"sampleCount": 63,
|
|
10
10
|
"limitations": "The declared 63-sample window is the prior alpha.2 review, not the newer cohort. Separate alpha.10 read-only profile aggregates on October 11 show Astra 3 selections across 18 conditional opportunities, summed expected selections 2.868. No Session IDs or sequence in aggregates. Bounded native metadata from the requested local Session shows five Sol turns followed by two Astra turns; the last has 178 steps under one frozen allocation. Tail scan is limited, not full history. Actual per-turn pool/weights are not durably retained. Two user-reported valid unknown classifier results are not a population unknown rate. No content, credentials or raw export uploaded."
|
|
11
11
|
},
|
|
12
|
-
"previousReview": "Prior
|
|
12
|
+
"previousReview": "Prior #332/#333/#334, alpha.11 #411 and alpha.12 #413. User clarified on October 11 that every eligible fitting model should retain positive opportunity across categories, not just within one category (#414). No newer live v3 outcome sample is available; retain #334.",
|
|
13
13
|
"findings": "Prior 63 decisions cannot attribute later incidents. Valid unknown is not a timeout. Projection regressions establish avoidable anchor loss/incidental-omission downgrade, not the actual task root cause. The reported Astra streak is two allocations, not 178 independent choices. Under an explicitly assumed two-model category pool, production Session/turn seeds reproduce both Astra choices at conditional shares 11.8% then 23.1%; this is conditional reproduction, not recovered decision evidence. A 20000-turn synthetic fixed-previous v1 scan yields 4563 Astra hits and 1014 adjacent pairs, showing runs are possible, not proving actual distribution conformity. No observed aggregate inflation is established. V1 intentionally grants marked continuity; the user now requests a stricter cost preference, not a statistically proven RNG repair.",
|
|
14
|
-
"decision": "
|
|
15
|
-
"validation": "
|
|
14
|
+
"decision": "High-cost-v4 replaces hard category filtering with positive per-model category factors: target 1, second 0.25, third 0.05 in the existing category order, unknown explicitly 0.02. Multiply by v3 ordinary/previous/high-cost/previous-high-cost weights 1/1.5/0.2/0.02. All hard-eligible fitting models participate; exact exclusions, current account/capability proof, input modality and headroom remain hard guards. These are uncalibrated biased trial coefficients, not optimal probabilities, quality evidence or fixed category quotas. Candidate population affects category totals. Intentionally permit non-target draws without task-failure inference, extra calls or in-turn switches. No-fit native recovery and auxiliary classifier policy remain unchanged. Capture candidate category/factor; preserve strict v1/v2/v3 historical explanations/cohorts and no-evidence opt-out. No new persistent population or measured quality/cost claim.",
|
|
15
|
+
"validation": "Five regression-first cross-category tests failed under v3. V3 and v4 actual selectors each ran 19 synthetic scenarios with 100000 Session/turn seeds per scenario. V4 retained positive observed opportunity for all fitting candidates; every expected/observed share differed by less than 0.5 percentage points. The six-model sequential v4 fixture selected marked Powerful 8577 times and recorded 93 adjacent marked pairs; ordinary Powerful 61215, Versatile 12705/12632, Lightweight 2410/2461. This is implementation evidence, not independent production probability or quality. Checked-in scripts/simulate-auto-distribution.mjs reproduces the matrix. Full pnpm verify passed: 2788 Vitest tests, 2 expected skips, 340 tooling tests, types/build/package smoke. Final exact-weight codec refinement passed 21 focused tests. Correct candidate-category aggregation, strict factor validation, legacy v1/v2/v3 codecs and evidence opt-out are covered. Native gateways and saved-mark/restart run in required CI, pending before merge. Local pack/tarball passed; recheck the final candidate after this review update. No real model calls, credentials, live metadata reads, profile changes or Desktop restart.",
|
|
16
16
|
"followUp": {
|
|
17
17
|
"owner": "Maintainer or Agent assigned to post-delivery review issue #334",
|
|
18
18
|
"trigger": "Review 200 new decisions with recorded reasons, or any further classifier capability/context/policy change; first inspect a successful unknown turn's projection and decoded-demand fields.",
|
|
19
|
-
"metric": "
|
|
20
|
-
"action": "Review
|
|
19
|
+
"metric": "By producer/policy and target/actual candidate category, compare selected/no-fit counts, opportunities and conditional expected selections. Split marked previous/base weights and population changes. Adjacent-turn rates require observed sequence, never daily aggregates. Track labeled outcomes of non-target selections before interpreting selection diversity as success, quality or savings. Legacy absence remains unknown.",
|
|
20
|
+
"action": "Review cross-category opportunity and task-quality regressions before retaining or tuning the declared factors; keep v1/v2/v3/v4 separate. No automatic parameter tuning, upload or live benchmark. Revert soft-category policy if reviewed outcomes reject the trial; retain genuine uncertainty and linked #334."
|
|
21
21
|
}
|
|
22
22
|
}
|
|
@@ -275,22 +275,28 @@ same-turn single-flight, eligibility and category policy remain unchanged.
|
|
|
275
275
|
| Unknown | Versatile | Versatile | Powerful |
|
|
276
276
|
|
|
277
277
|
These are declared routing policies, not benchmark rankings. Efficiency's
|
|
278
|
-
routine lightweight
|
|
278
|
+
routine lightweight target is an economical approximation, not proof of
|
|
279
279
|
equal output quality. Intelligence never treats short length alone as simple.
|
|
280
280
|
|
|
281
|
-
|
|
281
|
+
The target and remaining categories have this declared weight order:
|
|
282
282
|
powerful -> versatile -> lightweight; versatile -> powerful -> lightweight;
|
|
283
|
-
lightweight -> versatile -> powerful. Unclassified candidates
|
|
284
|
-
explicit
|
|
283
|
+
lightweight -> versatile -> powerful. Unclassified candidates retain a small
|
|
284
|
+
explicit opportunity, not fabricated categories. Unknown/new category values
|
|
285
285
|
remain unclassified with a bounded catalog diagnostic.
|
|
286
286
|
|
|
287
|
-
|
|
288
|
-
allocation
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
287
|
+
Across every eligible fitting category, use stable positive weighted Session/turn
|
|
288
|
+
allocation. The task/preference matrix is a preferred target, not exclusion.
|
|
289
|
+
Multiply per-model cost/recency weights by category factors 1 / 0.25 / 0.05
|
|
290
|
+
in the existing target/second/third order; unknown category uses 0.02 explicitly.
|
|
291
|
+
These factors are an uncalibrated trial. Ordinary weight 1 has finite
|
|
292
|
+
previous-model multiplier 1.5; high-cost base weight 0.2 decays tenfold to 0.02
|
|
293
|
+
only for the previous model. Every candidate keeps positive opportunity; a lone
|
|
294
|
+
candidate still has share 1. A different model's turn resets the penalty, without
|
|
295
|
+
extra streak state. Cost is independent of task difficulty and supplier
|
|
296
|
+
category; input fit does not prove task success. Non-target selection is
|
|
297
|
+
intentional weighted opportunity, not absence of a target candidate or proof
|
|
298
|
+
of task suitability. Never automatically escalate to establish whether
|
|
299
|
+
another model can solve the task.
|
|
294
300
|
These are reviewed policy parameters, not measured quality or prices. Once
|
|
295
301
|
admitted, all steps/retries share the decision. No supplier cache savings are
|
|
296
302
|
claimed. See [high-cost requirements](./auto-high-cost.md) and
|
|
@@ -303,7 +309,7 @@ never label this as a successful preference match or silently trim history.
|
|
|
303
309
|
## Explanation contract
|
|
304
310
|
|
|
305
311
|
Capture task source/demand/signals, assessment diagnostic, target/selected
|
|
306
|
-
category, fitting category count,
|
|
312
|
+
category, fitting cross-category count, target absence and continuity/tie-break or
|
|
307
313
|
no-fit selection at decision time. Client reads the explicit viewed Session and
|
|
308
314
|
turn through the existing strict Remote. It does not rerun classification or
|
|
309
315
|
consult today's picker. Show a decisive first sentence, bounded supporting
|
|
@@ -58,8 +58,10 @@ remains authoritative after final framing/attachment projection.
|
|
|
58
58
|
Task preference uses supplier `model_picker_category`, not capacity, model
|
|
59
59
|
names or reasoning effort as quality rankings. [Task policy](./auto-task-routing.md)
|
|
60
60
|
owns the demand matrix, default semantic assessment, category fallback,
|
|
61
|
-
cost-
|
|
62
|
-
|
|
61
|
+
high-cost one-turn recency decay, ordinary finite continuity and stable
|
|
62
|
+
positive-weight cross-category allocation. Every eligible fitting model has
|
|
63
|
+
positive opportunity; the task target biases per-model weights instead of
|
|
64
|
+
hard-filtering other categories. Unknown category stays explicit.
|
|
63
65
|
User-marked high cost is orthogonal to demand and category; see
|
|
64
66
|
[high-cost requirements](./auto-high-cost.md). Future policy changes must read
|
|
65
67
|
the [latest review](./auto-iteration-review.json) before implementation.
|