@goodandready/dsh-moa 0.2.73 → 0.2.74
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +9 -0
- package/README.ru.md +9 -0
- package/README.zh.md +9 -0
- package/lib/client.js +134 -2
- package/lib/events.js +2 -0
- package/lib/moa-schema.js +12 -0
- package/lib/pipeline/context.js +39 -2
- package/lib/pipeline/escalation.js +160 -0
- package/lib/pipeline/runner.js +59 -5
- package/lib/pipeline/self-consistency.js +135 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -186,6 +186,15 @@ Validates markdown code blocks across all candidate outputs locally using JavaSc
|
|
|
186
186
|
### 20. Early Exit upon Semantic Consensus
|
|
187
187
|
Compares normalized candidate proposals using token-level Sørensen-Dice similarity. When models arrive at identical or near-identical solutions (meeting or exceeding `early_exit_threshold`, default: `0.95`), the expensive Judge synthesis phase is automatically bypassed. The consensus solution is promoted directly to the workspace, saving up to 100% of judge token costs while accelerating pipeline latency. Configured via `early_exit_consensus_enabled: true` and `early_exit_threshold: 0.95`.
|
|
188
188
|
|
|
189
|
+
### 21. Dynamic Consensus Threshold & Multi-Judge Escalation
|
|
190
|
+
Evaluates pairwise candidate solution divergence (Disagreement Metric, 0.0–1.0) using normalized Sørensen-Dice similarity. When candidate consensus is weak (disagreement meets or exceeds `escalation_disagreement_threshold`, default: `0.40`), the pipeline automatically escalates:
|
|
191
|
+
- `auto_upgrade`: Upgrades from a lightweight or fast aggregator to a designated flagship senior judge (`escalation_judge` or first senior fallback judge).
|
|
192
|
+
- `council_majority`: Convenes a 3-judge panel with majority consensus voting.
|
|
193
|
+
Dispatches `moa/judge-escalation` Cordis events and displays escalation decisions in live stream telemetry. Configured via `escalation_policy: 'single' | 'auto_upgrade' | 'council_majority'`, `escalation_disagreement_threshold: 0.40`, and `escalation_judge`.
|
|
194
|
+
|
|
195
|
+
### 22. Self-Consistency Temperature Sampling for Single Models
|
|
196
|
+
Allows single-model configurations (such as a local Ollama/vLLM instance or a single API key) to leverage Mixture-of-Agents diversity. When enabled on a preset with 1 model slot, the runner expands it into $N$ parallel candidate streams with distinct temperatures (e.g. 0.2, 0.6, 0.9) and complementary personas (`proposer`, `challenger`, `minimalist`), calculates agreement ratio, and performs full MoA consensus synthesis. Dispatches `moa/self-consistency-expand` Cordis events. Configured via `self_consistency_enabled: true`, `self_consistency_samples: 3`, and `self_consistency_temperatures: [0.2, 0.6, 0.9]`.
|
|
197
|
+
|
|
189
198
|
### 6. Live Canvas 1-Click Preview (optional)
|
|
190
199
|
If `@goodandready/dsh-live-canvas` is installed in the same profile, `dsh-moa` pushes the promoted HTML file to the Live Canvas REST contract (`POST /dsh-live-canvas/api/preview`, served by the same harness webServer) and appends a one-click preview link (`/dsh-live-canvas/sandbox/<id>`) to the answer. Without the plugin the step is skipped silently — no errors in the log, no dead links.
|
|
191
200
|
|
package/README.ru.md
CHANGED
|
@@ -186,6 +186,15 @@ graph TD
|
|
|
186
186
|
### 20. Ранний выход по семантическому консенсусу
|
|
187
187
|
Сравнивает нормализованные решения кандидатов по токенному сходству Сёренсена-Дайса. Если решения моделей совпадают или практически идентичны (достигнут порог `early_exit_threshold`, по умолчанию 0.95), стадия вызова судьи (Judge) пропускается. Согласованный вариант немедленно продвигается в проект, экономя до 100% токенов судейской коллегии и сокращая задержку ответа. Управляется параметрами `early_exit_consensus_enabled: true` и `early_exit_threshold: 0.95`.
|
|
188
188
|
|
|
189
|
+
### 21. Динамический порог консенсуса и эскалация судейства (Judge Escalation)
|
|
190
|
+
Оценивает степень расхождения между решениями моделей-кандидатов (метрика Disagreement Metric, 0.0–1.0) на основе попарного расстояния Сёренсена-Дайса. Если расхождение превышает порог (`escalation_disagreement_threshold`, по умолчанию 0.40), пайплайн выполняет адаптивную эскалацию:
|
|
191
|
+
- `auto_upgrade`: автоматический переход с базового/быстрого судьи на флагманскую старшую модель (`escalation_judge` или первый доступный резервный судья).
|
|
192
|
+
- `council_majority`: созыв коллегии из 3 судей с голосованием большинством голосов.
|
|
193
|
+
Генерирует событие `moa/judge-escalation` и выводит статус эскалации в телеметрию стрима.
|
|
194
|
+
|
|
195
|
+
### 22. Самосогласованное температурное сэмплирование для одиночных моделей (Self-Consistency)
|
|
196
|
+
Позволяет пользователям с единственной моделью (локальная Ollama, vLLM или единственный API-ключ) получать преимущества ансамблевой валидации MoA. Автоматически разворачивает одиночный слот на $N$ параллельных запусков с различными температурами (`0.2, 0.6, 0.9`) и дополняющими друг друга инженерными персонами (`proposer`, `challenger`, `minimalist`), вычисляет коэффициент согласия (Self-Consistency Agreement) и синтезирует итоговый результат. Генерирует событие `moa/self-consistency-expand`.
|
|
197
|
+
|
|
189
198
|
### 6. Live Canvas 1-клик предпросмотр (опционально)
|
|
190
199
|
Если в профиле установлен `@goodandready/dsh-live-canvas`, `dsh-moa` отправляет промоученный HTML-файл в REST-контракт Live Canvas (`POST /dsh-live-canvas/api/preview`, тот же webServer харнесса) и добавляет к ответу ссылку на предпросмотр (`/dsh-live-canvas/sandbox/<id>`). Без плагина шаг пропускается тихо — без ошибок в журнале и без битых ссылок.
|
|
191
200
|
|
package/README.zh.md
CHANGED
|
@@ -163,6 +163,15 @@ graph TD
|
|
|
163
163
|
### 20. 语义共识提前退出机制 (Early Exit upon Semantic Consensus)
|
|
164
164
|
基于 Token 级 Sørensen-Dice 相似度算法比对归一化后的候选方案。当多个候选模型给出高度一致的代码方案(达到或超过 `early_exit_threshold` 阈值,默认 0.95)时,自动跳过高耗费的裁判综合评审阶段,直接将共识方案提升至项目,在大幅缩短响应延迟的同时节省多达 100% 的裁判 Token 支出。通过 `early_exit_consensus_enabled: true` 与 `early_exit_threshold: 0.95` 配置。
|
|
165
165
|
|
|
166
|
+
### 21. 动态共识阈值与裁判自适应升级 (Dynamic Consensus & Judge Escalation)
|
|
167
|
+
基于归一化 Sørensen-Dice 计算候选方案间的分歧度 (Disagreement Metric 0.0–1.0)。当候选分歧超过设定阈值 (`escalation_disagreement_threshold`,默认 0.40) 时,流水线自动触发升级策略:
|
|
168
|
+
- `auto_upgrade`: 自动将普通/快速裁判升级为指定的高阶旗舰裁判 (`escalation_judge` 或首选候补裁判)。
|
|
169
|
+
- `council_majority`: 组建 3 裁判多数席裁决团。
|
|
170
|
+
触发 `moa/judge-escalation` 事件并在实时流中呈现决策依据。
|
|
171
|
+
|
|
172
|
+
### 22. 单模型自洽性温度采样 (Self-Consistency Temperature Sampling)
|
|
173
|
+
专为仅配置单模型(如本地 Ollama/vLLM 或单一 API Key)的场景设计。开启后自动将单槽位扩展为 $N$ 个并行采样(分别采用 0.2、0.6、0.9 等递增温度与 `proposer`、`challenger`、`minimalist` 互补提示词),评估自洽一致性比率并进行全流程 MoA 综合聚合,触发 `moa/self-consistency-expand` 事件。
|
|
174
|
+
|
|
166
175
|
### 6. Live Canvas 一键预览(可选)
|
|
167
176
|
若同一 profile 中安装了 `@goodandready/dsh-live-canvas`,`dsh-moa` 会将提升后的 HTML 文件推送到 Live Canvas 的 REST 契约(`POST /dsh-live-canvas/api/preview`,由同一 harness webServer 提供服务),并在回答中附上一键预览链接(`/dsh-live-canvas/sandbox/<id>`)。未安装该插件时此步骤静默跳过——日志无报错,也不会出现死链接。
|
|
168
177
|
|
package/lib/client.js
CHANGED
|
@@ -213,7 +213,18 @@ window.__ModuleLoader__.load({
|
|
|
213
213
|
"workbench.promote_btn": "Promote Harmonized Files to Workspace",
|
|
214
214
|
"workbench.promoted_ok": "\u2705 Harmonized deliverable promoted to workspace root!",
|
|
215
215
|
"workbench.pros": "Pros:",
|
|
216
|
-
"workbench.cons": "Cons:"
|
|
216
|
+
"workbench.cons": "Cons:",
|
|
217
|
+
"aggregator.self_consistency_title": "\u{1F3B2} Self-Consistency Temperature Sampling (Single-Model Diversity)",
|
|
218
|
+
"aggregator.self_consistency_desc": "Expands a single reference model into multiple parallel samples with varied temperatures and personas, bringing MoA benefits to single-model setups.",
|
|
219
|
+
"aggregator.self_consistency_samples": "Samples:",
|
|
220
|
+
"aggregator.escalation_title": "\u2696\uFE0F Dynamic Consensus Threshold & Judge Escalation",
|
|
221
|
+
"aggregator.escalation_desc": "Detects divergence across candidates. When consensus is weak, automatically upgrades to a senior flagship judge or convenes a 3-judge panel.",
|
|
222
|
+
"aggregator.escalation_policy": "Escalation Policy:",
|
|
223
|
+
"aggregator.escalation_policy_single": "Standard Single Judge (No Escalation)",
|
|
224
|
+
"aggregator.escalation_policy_upgrade": "Auto-Upgrade to Senior Judge on Divergence",
|
|
225
|
+
"aggregator.escalation_policy_council": "Convene 3-Judge Majority Council on Divergence",
|
|
226
|
+
"aggregator.escalation_threshold": "Divergence Threshold:",
|
|
227
|
+
"aggregator.escalation_judge_label": "Senior Judge:"
|
|
217
228
|
};
|
|
218
229
|
const zh = {
|
|
219
230
|
title: "\u667A\u80FD\u4F53\u6DF7\u5408 (MoA)",
|
|
@@ -417,7 +428,18 @@ window.__ModuleLoader__.load({
|
|
|
417
428
|
"workbench.promote_btn": "\u63D0\u5347\u7C98\u5408\u7ED3\u679C\u81F3\u9879\u76EE\u6839\u76EE\u5F55",
|
|
418
429
|
"workbench.promoted_ok": "\u2705 \u7C98\u5408\u65B9\u6848\u5DF2\u6210\u529F\u63D0\u5347\u81F3\u9879\u76EE\u5DE5\u4F5C\u533A\uFF01",
|
|
419
430
|
"workbench.pros": "\u4F18\u52BF\uFF1A",
|
|
420
|
-
"workbench.cons": "\u4E0D\u8DB3\uFF1A"
|
|
431
|
+
"workbench.cons": "\u4E0D\u8DB3\uFF1A",
|
|
432
|
+
"aggregator.self_consistency_title": "\u{1F3B2} \u5355\u6A21\u578B\u81EA\u6D3D\u6027\u6E29\u5EA6\u91C7\u6837 (Self-Consistency)",
|
|
433
|
+
"aggregator.self_consistency_desc": "\u5728\u4EC5\u914D\u7F6E\u5355\u6A21\u578B\u65F6\uFF0C\u751F\u6210\u4E0D\u540C\u6E29\u5EA6\u4E0E\u89D2\u8272\u7684\u591A\u4E2A\u5E76\u884C\u5019\u9009\u65B9\u6848\uFF0C\u8BA9\u672C\u5730 Ollama \u6216\u5355 Key \u4E5F\u4EAB\u6709 MoA \u805A\u5408\u4F18\u52BF\u3002",
|
|
434
|
+
"aggregator.self_consistency_samples": "\u91C7\u6837\u6570:",
|
|
435
|
+
"aggregator.escalation_title": "\u2696\uFE0F \u52A8\u6001\u5171\u8BC6\u9608\u503C\u4E0E\u88C1\u5224\u5347\u7EA7 (Judge Escalation)",
|
|
436
|
+
"aggregator.escalation_desc": "\u68C0\u6D4B\u5019\u9009\u65B9\u6848\u95F4\u7684\u5206\u6B67\u5EA6\u3002\u5F53\u5171\u8BC6\u8F83\u5F31\u65F6\uFF0C\u81EA\u52A8\u5347\u7EA7\u81F3\u9AD8\u9636\u65D7\u8230\u88C1\u5224\u6216\u7EC4\u5EFA 3 \u88C1\u5224\u591A\u6570\u5E2D\u3002",
|
|
437
|
+
"aggregator.escalation_policy": "\u5347\u7EA7\u7B56\u7565:",
|
|
438
|
+
"aggregator.escalation_policy_single": "\u6807\u51C6\u5355\u88C1\u5224 (\u4E0D\u5347\u7EA7)",
|
|
439
|
+
"aggregator.escalation_policy_upgrade": "\u5206\u6B67\u65F6\u81EA\u52A8\u5347\u7EA7\u81F3\u9AD8\u9636\u88C1\u5224",
|
|
440
|
+
"aggregator.escalation_policy_council": "\u5206\u6B67\u65F6\u7EC4\u5EFA 3 \u88C1\u5224\u591A\u6570\u51B3\u7B56\u5E2D",
|
|
441
|
+
"aggregator.escalation_threshold": "\u5206\u6B67\u9608\u503C:",
|
|
442
|
+
"aggregator.escalation_judge_label": "\u9AD8\u9636\u88C1\u5224\u6A21\u578B:"
|
|
421
443
|
};
|
|
422
444
|
function makeT(dict, fallback) {
|
|
423
445
|
return function t(key, vars) {
|
|
@@ -2427,6 +2449,116 @@ window.__ModuleLoader__.load({
|
|
|
2427
2449
|
t("aggregator.semantic_cache_desc") || "Persists intermediate candidate outputs locally by deterministic SHA-256 fingerprint, eliminating duplicate API costs on retry or repeat requests."
|
|
2428
2450
|
)
|
|
2429
2451
|
),
|
|
2452
|
+
/* Self-Consistency Temperature Sampling (#301) */
|
|
2453
|
+
React.createElement(
|
|
2454
|
+
"div",
|
|
2455
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6, padding: "8px 12px", background: "var(--dsw-alias-bg-layer-2)", borderRadius: 8, border: "1px solid var(--dsw-alias-border-l2)", marginTop: 8 } },
|
|
2456
|
+
React.createElement(
|
|
2457
|
+
"label",
|
|
2458
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, cursor: "pointer", fontSize: 13, fontWeight: 500 } },
|
|
2459
|
+
React.createElement("input", {
|
|
2460
|
+
type: "checkbox",
|
|
2461
|
+
checked: Boolean(currentPreset.self_consistency_enabled),
|
|
2462
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, self_consistency_enabled: e.target.checked })),
|
|
2463
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2464
|
+
}),
|
|
2465
|
+
t("aggregator.self_consistency_title") || "\u{1F3B2} Self-Consistency Temperature Sampling (Single-Model Diversity)"
|
|
2466
|
+
),
|
|
2467
|
+
React.createElement(
|
|
2468
|
+
"div",
|
|
2469
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2470
|
+
t("aggregator.self_consistency_desc") || "Expands a single reference model into multiple parallel samples with varied temperatures and personas, bringing MoA benefits to single-model setups."
|
|
2471
|
+
),
|
|
2472
|
+
currentPreset.self_consistency_enabled && React.createElement(
|
|
2473
|
+
"div",
|
|
2474
|
+
{ style: { marginLeft: 22, marginTop: 4, display: "flex", alignItems: "center", gap: 12 } },
|
|
2475
|
+
React.createElement("span", { style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)" } }, (t("aggregator.self_consistency_samples") || "Samples:") + " " + (currentPreset.self_consistency_samples || 3)),
|
|
2476
|
+
React.createElement("input", {
|
|
2477
|
+
type: "range",
|
|
2478
|
+
min: 2,
|
|
2479
|
+
max: 5,
|
|
2480
|
+
value: currentPreset.self_consistency_samples || 3,
|
|
2481
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, self_consistency_samples: Number(e.target.value) })),
|
|
2482
|
+
style: { width: 100, accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))" }
|
|
2483
|
+
})
|
|
2484
|
+
)
|
|
2485
|
+
),
|
|
2486
|
+
/* Dynamic Consensus Threshold & Multi-Judge Escalation (#300) */
|
|
2487
|
+
React.createElement(
|
|
2488
|
+
"div",
|
|
2489
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6, padding: "8px 12px", background: "var(--dsw-alias-bg-layer-2)", borderRadius: 8, border: "1px solid var(--dsw-alias-border-l2)", marginTop: 8 } },
|
|
2490
|
+
React.createElement(
|
|
2491
|
+
"div",
|
|
2492
|
+
{ style: { fontSize: 13, fontWeight: 600, color: "var(--dsw-alias-label-primary)" } },
|
|
2493
|
+
t("aggregator.escalation_title") || "\u2696\uFE0F Dynamic Consensus Threshold & Judge Escalation"
|
|
2494
|
+
),
|
|
2495
|
+
React.createElement(
|
|
2496
|
+
"div",
|
|
2497
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)" } },
|
|
2498
|
+
t("aggregator.escalation_desc") || "Detects divergence across candidates. When consensus is weak, automatically upgrades to a senior flagship judge or convenes a 3-judge panel."
|
|
2499
|
+
),
|
|
2500
|
+
React.createElement(
|
|
2501
|
+
"div",
|
|
2502
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, marginTop: 4 } },
|
|
2503
|
+
React.createElement("span", { style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)" } }, t("aggregator.escalation_policy") || "Escalation Policy:"),
|
|
2504
|
+
React.createElement(
|
|
2505
|
+
"select",
|
|
2506
|
+
{
|
|
2507
|
+
value: currentPreset.escalation_policy || "single",
|
|
2508
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, escalation_policy: e.target.value })),
|
|
2509
|
+
style: {
|
|
2510
|
+
padding: "4px 8px",
|
|
2511
|
+
fontSize: 12,
|
|
2512
|
+
borderRadius: 6,
|
|
2513
|
+
border: "1px solid var(--dsw-alias-border-l2)",
|
|
2514
|
+
background: "var(--dsw-alias-bg-layer-1)",
|
|
2515
|
+
color: "var(--dsw-alias-label-primary)"
|
|
2516
|
+
}
|
|
2517
|
+
},
|
|
2518
|
+
React.createElement("option", { value: "single" }, t("aggregator.escalation_policy_single") || "Standard Single Judge (No Escalation)"),
|
|
2519
|
+
React.createElement("option", { value: "auto_upgrade" }, t("aggregator.escalation_policy_upgrade") || "Auto-Upgrade to Senior Judge on Divergence"),
|
|
2520
|
+
React.createElement("option", { value: "council_majority" }, t("aggregator.escalation_policy_council") || "Convene 3-Judge Majority Council on Divergence")
|
|
2521
|
+
)
|
|
2522
|
+
),
|
|
2523
|
+
currentPreset.escalation_policy && currentPreset.escalation_policy !== "single" && React.createElement(
|
|
2524
|
+
"div",
|
|
2525
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6, marginTop: 4 } },
|
|
2526
|
+
React.createElement(
|
|
2527
|
+
"div",
|
|
2528
|
+
{ style: { display: "flex", alignItems: "center", gap: 12 } },
|
|
2529
|
+
React.createElement("span", { style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)" } }, (t("aggregator.escalation_threshold") || "Divergence Threshold:") + " " + Math.round((currentPreset.escalation_disagreement_threshold ?? 0.4) * 100) + "%"),
|
|
2530
|
+
React.createElement("input", {
|
|
2531
|
+
type: "range",
|
|
2532
|
+
min: 10,
|
|
2533
|
+
max: 90,
|
|
2534
|
+
value: Math.round((currentPreset.escalation_disagreement_threshold ?? 0.4) * 100),
|
|
2535
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, escalation_disagreement_threshold: Number(e.target.value) / 100 })),
|
|
2536
|
+
style: { width: 120, accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))" }
|
|
2537
|
+
})
|
|
2538
|
+
),
|
|
2539
|
+
currentPreset.escalation_policy === "auto_upgrade" && React.createElement(
|
|
2540
|
+
"div",
|
|
2541
|
+
{ style: { display: "flex", alignItems: "center", gap: 8 } },
|
|
2542
|
+
React.createElement("span", { style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)" } }, t("aggregator.escalation_judge_label") || "Senior Judge:"),
|
|
2543
|
+
React.createElement(
|
|
2544
|
+
"div",
|
|
2545
|
+
{ style: { flex: 1, maxWidth: 300 } },
|
|
2546
|
+
React.createElement(SearchableModelPicker, {
|
|
2547
|
+
provider: currentPreset.escalation_judge?.provider || "",
|
|
2548
|
+
model: currentPreset.escalation_judge?.model || "",
|
|
2549
|
+
onChange: (prov, mod) => {
|
|
2550
|
+
updateCurrentPreset((p) => ({
|
|
2551
|
+
...p,
|
|
2552
|
+
escalation_judge: { provider: prov, model: mod }
|
|
2553
|
+
}));
|
|
2554
|
+
},
|
|
2555
|
+
availableModels,
|
|
2556
|
+
t
|
|
2557
|
+
})
|
|
2558
|
+
)
|
|
2559
|
+
)
|
|
2560
|
+
)
|
|
2561
|
+
),
|
|
2430
2562
|
/* Fallback Judges Chain */
|
|
2431
2563
|
React.createElement(
|
|
2432
2564
|
"div",
|
package/lib/events.js
CHANGED
|
@@ -7,6 +7,8 @@ export const MOA_EVENTS = {
|
|
|
7
7
|
CANDIDATE_COMPLETE: 'moa/candidate-complete',
|
|
8
8
|
SYNTAX_VALIDATION: 'moa/syntax-validation',
|
|
9
9
|
EARLY_CONSENSUS: 'moa/early-consensus',
|
|
10
|
+
JUDGE_ESCALATION: 'moa/judge-escalation',
|
|
11
|
+
SELF_CONSISTENCY_EXPAND: 'moa/self-consistency-expand',
|
|
10
12
|
CRITIQUE_COMPLETE: 'moa/critique-complete',
|
|
11
13
|
JUDGE_VERDICT: 'moa/judge-verdict',
|
|
12
14
|
CONTEXT_PRUNING: 'moa/context-pruning',
|
package/lib/moa-schema.js
CHANGED
|
@@ -83,6 +83,12 @@ export const PresetSchema = z.object({
|
|
|
83
83
|
context_pruning_diff_lines: z.number().default(5),
|
|
84
84
|
semantic_cache_enabled: z.boolean().default(false),
|
|
85
85
|
semantic_cache_ttl_hours: z.number().default(168),
|
|
86
|
+
self_consistency_enabled: z.boolean().default(false),
|
|
87
|
+
self_consistency_samples: z.number().default(3),
|
|
88
|
+
self_consistency_temperatures: z.array(z.number()).default([0.2, 0.6, 0.9]),
|
|
89
|
+
escalation_policy: z.string().default('single'),
|
|
90
|
+
escalation_disagreement_threshold: z.number().default(0.40),
|
|
91
|
+
escalation_judge: ModelSlotSchema.default({ provider: '', model: '' }),
|
|
86
92
|
})
|
|
87
93
|
|
|
88
94
|
export const Config = z.object({
|
|
@@ -112,6 +118,12 @@ export const Config = z.object({
|
|
|
112
118
|
context_pruning_diff_lines: z.number().default(5).volatile(),
|
|
113
119
|
semantic_cache_enabled: z.boolean().default(false).volatile(),
|
|
114
120
|
semantic_cache_ttl_hours: z.number().default(168).volatile(),
|
|
121
|
+
self_consistency_enabled: z.boolean().default(false).volatile(),
|
|
122
|
+
self_consistency_samples: z.number().default(3).volatile(),
|
|
123
|
+
self_consistency_temperatures: z.array(z.number()).default([0.2, 0.6, 0.9]).volatile(),
|
|
124
|
+
escalation_policy: z.string().default('single').volatile(),
|
|
125
|
+
escalation_disagreement_threshold: z.number().default(0.40).volatile(),
|
|
126
|
+
escalation_judge: ModelSlotSchema.default({ provider: '', model: '' }).volatile(),
|
|
115
127
|
smart_routing_model: ModelSlotSchema.default({ provider: '', model: '' }).volatile(),
|
|
116
128
|
prices: z.dict(PriceRow).default({}).volatile(),
|
|
117
129
|
presets: z.array(PresetSchema).default([
|
package/lib/pipeline/context.js
CHANGED
|
@@ -1,3 +1,5 @@
|
|
|
1
|
+
import { expandSelfConsistencySlots } from './self-consistency.js'
|
|
2
|
+
import { emitEvent, MOA_EVENTS } from '../events.js'
|
|
1
3
|
import { balanceCandidateQuotas } from '../moa-quota-balancer.js'
|
|
2
4
|
import { collectProjectContext, formatProjectContext, isRefinementTask } from '../workspace/scanner.js'
|
|
3
5
|
import { applyBudgetGuardrails, createBudgetAbortPayload } from '../moa-budget.js'
|
|
@@ -27,7 +29,40 @@ export async function stageProjectContext({
|
|
|
27
29
|
const refTemp = typeof preset?.reference_temperature === 'number' ? preset.reference_temperature : 0.6
|
|
28
30
|
const aggTemp = typeof preset?.aggregator_temperature === 'number' ? preset.aggregator_temperature : 0.4
|
|
29
31
|
const judgeCriteria = preset?.judge_criteria || ''
|
|
30
|
-
|
|
32
|
+
const selfConsistencyEnabled = Boolean(preset?.self_consistency_enabled)
|
|
33
|
+
const selfConsistencySamples = typeof preset?.self_consistency_samples === 'number' ? preset.self_consistency_samples : 3
|
|
34
|
+
const selfConsistencyTemperatures = Array.isArray(preset?.self_consistency_temperatures)
|
|
35
|
+
? preset.self_consistency_temperatures
|
|
36
|
+
: [0.2, 0.6, 0.9]
|
|
37
|
+
const escalationPolicy = preset?.escalation_policy || 'single'
|
|
38
|
+
const escalationDisagreementThreshold = typeof preset?.escalation_disagreement_threshold === 'number'
|
|
39
|
+
? preset.escalation_disagreement_threshold
|
|
40
|
+
: 0.40
|
|
41
|
+
const escalationJudge = (preset?.escalation_judge?.model) ? preset.escalation_judge : null
|
|
42
|
+
|
|
43
|
+
let effectiveReferenceModels = referenceModels
|
|
44
|
+
let selfConsistencyExpanded = false
|
|
45
|
+
if (selfConsistencyEnabled) {
|
|
46
|
+
const scRes = expandSelfConsistencySlots(referenceModels, {
|
|
47
|
+
enabled: true,
|
|
48
|
+
samples: selfConsistencySamples,
|
|
49
|
+
temperatures: selfConsistencyTemperatures,
|
|
50
|
+
})
|
|
51
|
+
if (scRes.isExpanded) {
|
|
52
|
+
effectiveReferenceModels = scRes.references
|
|
53
|
+
selfConsistencyExpanded = true
|
|
54
|
+
emitEvent(ctx, MOA_EVENTS.SELF_CONSISTENCY_EXPAND, {
|
|
55
|
+
originalModel: referenceModels[0]?.model,
|
|
56
|
+
sampleCount: scRes.sampleCount,
|
|
57
|
+
temperatures: selfConsistencyTemperatures.slice(0, scRes.sampleCount),
|
|
58
|
+
})
|
|
59
|
+
if (typeof onProgress === 'function') {
|
|
60
|
+
onProgress(`🎲 *Self-Consistency: Expanded single model into ${scRes.sampleCount} diverse temperature samples (${selfConsistencyTemperatures.slice(0, scRes.sampleCount).join(', ')})*\n`)
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
let isFastMode = effectiveReferenceModels.length === 1 && !preset?.curator_synthesis
|
|
31
66
|
const isCuratorSynthesis = Boolean(preset?.curator_synthesis)
|
|
32
67
|
const isStreamAggregator = preset?.stream_aggregator !== false
|
|
33
68
|
const isQuorumEnabled = Boolean(preset?.quorum_enabled)
|
|
@@ -65,7 +100,7 @@ export async function stageProjectContext({
|
|
|
65
100
|
}
|
|
66
101
|
|
|
67
102
|
const budgetCheck = applyBudgetGuardrails({
|
|
68
|
-
references:
|
|
103
|
+
references: effectiveReferenceModels,
|
|
69
104
|
enabled: preset?.budget_guard_enabled,
|
|
70
105
|
maxBudgetUsd: preset?.max_budget_usd,
|
|
71
106
|
action: preset?.budget_action || 'trim',
|
|
@@ -133,6 +168,8 @@ export async function stageProjectContext({
|
|
|
133
168
|
localPreflightCheck, earlyExitConsensusEnabled, earlyExitThreshold,
|
|
134
169
|
contextPruningEnabled, contextPruningDiffLines,
|
|
135
170
|
semanticCacheEnabled, semanticCacheTtlHours,
|
|
171
|
+
selfConsistencyEnabled, selfConsistencyExpanded, selfConsistencySamples, selfConsistencyTemperatures,
|
|
172
|
+
escalationPolicy, escalationDisagreementThreshold, escalationJudge,
|
|
136
173
|
}
|
|
137
174
|
}
|
|
138
175
|
|
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Dynamic Consensus Threshold & Multi-Judge Escalation (#300)
|
|
3
|
+
*
|
|
4
|
+
* Detects divergence among candidate outputs (Disagreement Metric).
|
|
5
|
+
* When candidate consensus is weak or contradictory, automatically escalates:
|
|
6
|
+
* - auto_upgrade: upgrades from lightweight/fast judge to flagship/senior judge
|
|
7
|
+
* - council_majority: convenes a 3-judge panel with majority voting
|
|
8
|
+
*/
|
|
9
|
+
|
|
10
|
+
import { calculateSimilarity } from './early-consensus.js'
|
|
11
|
+
|
|
12
|
+
/**
|
|
13
|
+
* Calculates the Disagreement Metric (0.0 to 1.0) across candidate outputs.
|
|
14
|
+
* 0.0 means perfect consensus (identical solutions).
|
|
15
|
+
* 1.0 means total divergence (completely distinct solutions).
|
|
16
|
+
*
|
|
17
|
+
* @param {Array<object>} candidateOutputs
|
|
18
|
+
* @returns {number} Disagreement metric between 0.0 and 1.0
|
|
19
|
+
*/
|
|
20
|
+
export function calculateDisagreementMetric(candidateOutputs = []) {
|
|
21
|
+
if (!Array.isArray(candidateOutputs) || candidateOutputs.length < 2) {
|
|
22
|
+
return 0.0
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
const validTexts = candidateOutputs
|
|
26
|
+
.filter((c) => c && typeof c.text === 'string' && c.text.trim())
|
|
27
|
+
.map((c) => c.text)
|
|
28
|
+
|
|
29
|
+
if (validTexts.length < 2) return 0.0
|
|
30
|
+
|
|
31
|
+
let totalSim = 0
|
|
32
|
+
let pairs = 0
|
|
33
|
+
|
|
34
|
+
for (let i = 0; i < validTexts.length; i++) {
|
|
35
|
+
for (let j = i + 1; j < validTexts.length; j++) {
|
|
36
|
+
totalSim += calculateSimilarity(validTexts[i], validTexts[j])
|
|
37
|
+
pairs++
|
|
38
|
+
}
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
if (pairs === 0) return 0.0
|
|
42
|
+
const avgSim = totalSim / pairs
|
|
43
|
+
const disagreement = Math.max(0.0, Math.min(1.0, 1.0 - avgSim))
|
|
44
|
+
return Number(disagreement.toFixed(4))
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
/**
|
|
48
|
+
* Evaluates whether candidate divergence warrants judge escalation.
|
|
49
|
+
*
|
|
50
|
+
* @param {object} params
|
|
51
|
+
* @param {Array<object>} params.candidateOutputs - Array of successful candidate results
|
|
52
|
+
* @param {object} params.primaryJudge - Primary configured judge slot
|
|
53
|
+
* @param {Array<object>} [params.fallbackJudges=[]] - Chain of backup judges
|
|
54
|
+
* @param {object} [params.escalationJudge] - Explicit flagship upgrade judge
|
|
55
|
+
* @param {string} [params.policy='single'] - 'single' | 'auto_upgrade' | 'council_majority'
|
|
56
|
+
* @param {number} [params.threshold=0.40] - Disagreement threshold to trigger escalation
|
|
57
|
+
* @returns {object} Escalation decision and effective judge(s)
|
|
58
|
+
*/
|
|
59
|
+
export function evaluateJudgeEscalation({
|
|
60
|
+
candidateOutputs = [],
|
|
61
|
+
primaryJudge = { provider: '', model: '' },
|
|
62
|
+
fallbackJudges = [],
|
|
63
|
+
escalationJudge = null,
|
|
64
|
+
policy = 'single',
|
|
65
|
+
threshold = 0.40,
|
|
66
|
+
} = {}) {
|
|
67
|
+
const normPolicy = String(policy || 'single').toLowerCase()
|
|
68
|
+
|
|
69
|
+
if (normPolicy === 'single' || !Array.isArray(candidateOutputs) || candidateOutputs.length < 2) {
|
|
70
|
+
return {
|
|
71
|
+
escalated: false,
|
|
72
|
+
escalatedType: 'none',
|
|
73
|
+
judge: primaryJudge,
|
|
74
|
+
councilJudges: [primaryJudge],
|
|
75
|
+
disagreement: 0.0,
|
|
76
|
+
threshold,
|
|
77
|
+
reason: 'Standard single judge evaluation',
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
const disagreement = calculateDisagreementMetric(candidateOutputs)
|
|
82
|
+
const isTriggered = disagreement >= threshold
|
|
83
|
+
|
|
84
|
+
if (!isTriggered) {
|
|
85
|
+
return {
|
|
86
|
+
escalated: false,
|
|
87
|
+
escalatedType: 'none',
|
|
88
|
+
judge: primaryJudge,
|
|
89
|
+
councilJudges: [primaryJudge],
|
|
90
|
+
disagreement,
|
|
91
|
+
threshold,
|
|
92
|
+
reason: `Consensus adequate: disagreement ${(disagreement * 100).toFixed(1)}% < threshold ${(threshold * 100).toFixed(1)}%`,
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
if (normPolicy === 'auto_upgrade') {
|
|
97
|
+
// Pick designated escalationJudge, or the first available fallbackJudge
|
|
98
|
+
let targetJudge = null
|
|
99
|
+
if (escalationJudge && escalationJudge.model) {
|
|
100
|
+
targetJudge = escalationJudge
|
|
101
|
+
} else if (Array.isArray(fallbackJudges) && fallbackJudges.length > 0 && fallbackJudges[0]?.model) {
|
|
102
|
+
targetJudge = fallbackJudges[0]
|
|
103
|
+
} else {
|
|
104
|
+
targetJudge = primaryJudge
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
const upgraded = targetJudge.model !== primaryJudge.model || targetJudge.provider !== primaryJudge.provider
|
|
108
|
+
return {
|
|
109
|
+
escalated: upgraded,
|
|
110
|
+
escalatedType: upgraded ? 'auto_upgrade' : 'none',
|
|
111
|
+
judge: targetJudge,
|
|
112
|
+
originalJudge: primaryJudge,
|
|
113
|
+
councilJudges: [targetJudge],
|
|
114
|
+
disagreement,
|
|
115
|
+
threshold,
|
|
116
|
+
reason: upgraded
|
|
117
|
+
? `High candidate divergence (${(disagreement * 100).toFixed(1)}% >= ${(threshold * 100).toFixed(1)}%): upgraded judge to ${targetJudge.provider ? targetJudge.provider + ':' : ''}${targetJudge.model}`
|
|
118
|
+
: `High divergence detected (${(disagreement * 100).toFixed(1)}%), but no senior fallback judge configured`,
|
|
119
|
+
}
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
if (normPolicy === 'council_majority') {
|
|
123
|
+
const council = [primaryJudge]
|
|
124
|
+
if (escalationJudge && escalationJudge.model && escalationJudge.model !== primaryJudge.model) {
|
|
125
|
+
council.push(escalationJudge)
|
|
126
|
+
}
|
|
127
|
+
if (Array.isArray(fallbackJudges)) {
|
|
128
|
+
for (const fb of fallbackJudges) {
|
|
129
|
+
if (council.length >= 3) break
|
|
130
|
+
if (fb && fb.model && !council.some((c) => c.model === fb.model && c.provider === fb.provider)) {
|
|
131
|
+
council.push(fb)
|
|
132
|
+
}
|
|
133
|
+
}
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
const hasMulti = council.length > 1
|
|
137
|
+
return {
|
|
138
|
+
escalated: hasMulti,
|
|
139
|
+
escalatedType: hasMulti ? 'council_majority' : 'none',
|
|
140
|
+
judge: council[0],
|
|
141
|
+
originalJudge: primaryJudge,
|
|
142
|
+
councilJudges: council,
|
|
143
|
+
disagreement,
|
|
144
|
+
threshold,
|
|
145
|
+
reason: hasMulti
|
|
146
|
+
? `High candidate divergence (${(disagreement * 100).toFixed(1)}% >= ${(threshold * 100).toFixed(1)}%): assembled ${council.length}-judge council`
|
|
147
|
+
: `High divergence detected, but fewer than 2 judges available for council`,
|
|
148
|
+
}
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
return {
|
|
152
|
+
escalated: false,
|
|
153
|
+
escalatedType: 'none',
|
|
154
|
+
judge: primaryJudge,
|
|
155
|
+
councilJudges: [primaryJudge],
|
|
156
|
+
disagreement,
|
|
157
|
+
threshold,
|
|
158
|
+
reason: 'Standard single judge',
|
|
159
|
+
}
|
|
160
|
+
}
|
package/lib/pipeline/runner.js
CHANGED
|
@@ -1,3 +1,5 @@
|
|
|
1
|
+
import { evaluateJudgeEscalation } from './escalation.js'
|
|
2
|
+
import { calculateSelfConsistencyAgreement } from './self-consistency.js'
|
|
1
3
|
import { pruneCandidateEnsemble, formatPruningDirective } from './context-pruner.js'
|
|
2
4
|
import { buildAstCompositeMergeDirective } from '../moa-ast-merge.js'
|
|
3
5
|
import { runVisualMoaConsensus } from '../moa-visual-consensus.js'
|
|
@@ -106,6 +108,13 @@ export async function runMoAPipeline({
|
|
|
106
108
|
})
|
|
107
109
|
const syntaxDirective = syntaxResult.filterApplied ? formatSyntaxDirective(referenceOutputs, { blind: ctxStage.isBlindEvaluation }) : null
|
|
108
110
|
|
|
111
|
+
if (ctxStage.selfConsistencyExpanded && successfulRefs.length > 1) {
|
|
112
|
+
const scAgreement = calculateSelfConsistencyAgreement(successfulRefs)
|
|
113
|
+
if (typeof onProgress === 'function') {
|
|
114
|
+
onProgress(`🎯 *Self-Consistency Agreement: ${(scAgreement.agreementRatio * 100).toFixed(0)}% (${scAgreement.isStrongConsensus ? 'Strong consensus' : 'Divergent sampling'}) across ${scAgreement.sampleCount} samples*\n`)
|
|
115
|
+
}
|
|
116
|
+
}
|
|
117
|
+
|
|
109
118
|
// Stage 3: Clarifying Questions Branch
|
|
110
119
|
if (ctxStage.needsQuestions) {
|
|
111
120
|
return await stageClarifyingQuestions({
|
|
@@ -190,11 +199,56 @@ export async function runMoAPipeline({
|
|
|
190
199
|
return { error: new Error('Turn aborted') }
|
|
191
200
|
}
|
|
192
201
|
|
|
202
|
+
// Pre-Stage 6: Dynamic Consensus Threshold & Multi-Judge Escalation (#300)
|
|
203
|
+
const escalationResult = evaluateJudgeEscalation({
|
|
204
|
+
candidateOutputs: syntaxResult.validCandidates.length > 0 ? syntaxResult.validCandidates : successfulRefs,
|
|
205
|
+
primaryJudge: ctxStage.primaryJudge,
|
|
206
|
+
fallbackJudges: ctxStage.fallbackJudges,
|
|
207
|
+
escalationJudge: ctxStage.escalationJudge,
|
|
208
|
+
policy: ctxStage.escalationPolicy,
|
|
209
|
+
threshold: ctxStage.escalationDisagreementThreshold,
|
|
210
|
+
})
|
|
211
|
+
|
|
212
|
+
let effectivePrimaryJudge = ctxStage.primaryJudge
|
|
213
|
+
let effectiveFallbackJudges = ctxStage.fallbackJudges
|
|
214
|
+
let effectiveJudgesChain = ctxStage.judgesChain
|
|
215
|
+
let effectivePreset = preset
|
|
216
|
+
|
|
217
|
+
if (escalationResult.escalated) {
|
|
218
|
+
emitEvent(ctx, MOA_EVENTS.JUDGE_ESCALATION, {
|
|
219
|
+
runId,
|
|
220
|
+
policy: ctxStage.escalationPolicy,
|
|
221
|
+
disagreement: escalationResult.disagreement,
|
|
222
|
+
threshold: escalationResult.threshold,
|
|
223
|
+
escalatedType: escalationResult.escalatedType,
|
|
224
|
+
originalJudge: escalationResult.originalJudge,
|
|
225
|
+
escalatedJudge: escalationResult.judge,
|
|
226
|
+
councilJudges: escalationResult.councilJudges,
|
|
227
|
+
})
|
|
228
|
+
if (typeof onProgress === 'function') {
|
|
229
|
+
onProgress(`⚖️ *Judge Escalation (${escalationResult.escalatedType}): ${escalationResult.reason}*\n`)
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
if (escalationResult.escalatedType === 'auto_upgrade') {
|
|
233
|
+
effectivePrimaryJudge = escalationResult.judge
|
|
234
|
+
effectiveFallbackJudges = ctxStage.fallbackJudges.filter(
|
|
235
|
+
(j) => !(j.provider === escalationResult.judge.provider && j.model === escalationResult.judge.model)
|
|
236
|
+
)
|
|
237
|
+
effectiveJudgesChain = [effectivePrimaryJudge, ...effectiveFallbackJudges]
|
|
238
|
+
} else if (escalationResult.escalatedType === 'council_majority') {
|
|
239
|
+
effectivePreset = {
|
|
240
|
+
...preset,
|
|
241
|
+
multi_judge_enabled: true,
|
|
242
|
+
judge_models: escalationResult.councilJudges,
|
|
243
|
+
}
|
|
244
|
+
}
|
|
245
|
+
}
|
|
246
|
+
|
|
193
247
|
// Stage 6: Multi-Judge Panel
|
|
194
248
|
const mjRes = await stageMultiJudge({
|
|
195
|
-
preset,
|
|
249
|
+
preset: effectivePreset,
|
|
196
250
|
successfulRefs: syntaxResult.validCandidates.length > 0 ? syntaxResult.validCandidates : successfulRefs,
|
|
197
|
-
referenceOutputs, primaryJudge:
|
|
251
|
+
referenceOutputs, primaryJudge: effectivePrimaryJudge,
|
|
198
252
|
userPrompt, judgeCriteria: ctxStage.judgeCriteria, prices, maxTokens: ctxStage.maxTokens,
|
|
199
253
|
callLlm, signal, onProgress, cwd, runId, isRefinement: ctxStage.isRefinement,
|
|
200
254
|
isFastMode: ctxStage.isFastMode, collectedCtx: ctxStage.collectedCtx, startTime,
|
|
@@ -263,9 +317,9 @@ export async function runMoAPipeline({
|
|
|
263
317
|
judgeCriteria: ctxStage.judgeCriteria,
|
|
264
318
|
isCuratorSynthesis: ctxStage.isCuratorSynthesis, isBlindEvaluation: ctxStage.isBlindEvaluation,
|
|
265
319
|
priorTurnBaseline: ctxStage.priorTurnBaseline, compositeMergeDirective, fileScoringDirective,
|
|
266
|
-
multiJudgeResult, preset, prices, maxTokens: ctxStage.maxTokens, isFastMode: ctxStage.isFastMode,
|
|
267
|
-
primaryJudge:
|
|
268
|
-
judgesChain:
|
|
320
|
+
multiJudgeResult, preset: effectivePreset, prices, maxTokens: ctxStage.maxTokens, isFastMode: ctxStage.isFastMode,
|
|
321
|
+
primaryJudge: effectivePrimaryJudge, fallbackJudges: effectiveFallbackJudges,
|
|
322
|
+
judgesChain: effectiveJudgesChain, aggTemp: ctxStage.aggTemp,
|
|
269
323
|
aggTimeoutSec: ctxStage.aggTimeoutSec, isStreamAggregator: ctxStage.isStreamAggregator,
|
|
270
324
|
onStreamDelta, signal, callLlm, onProgress, cwd, runId, isRefinement: ctxStage.isRefinement,
|
|
271
325
|
collectedCtx: ctxStage.collectedCtx, startTime,
|
|
@@ -0,0 +1,135 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Self-Consistency Temperature Sampling for Single Models (#301)
|
|
3
|
+
*
|
|
4
|
+
* Allows users with a single model slot (e.g., local Ollama/vLLM or a single API key)
|
|
5
|
+
* to benefit from Mixture of Agents ensemble diversity by generating parallel samples
|
|
6
|
+
* at varying temperatures and complementary system personas, followed by consistency voting.
|
|
7
|
+
*/
|
|
8
|
+
|
|
9
|
+
import { ROLE_PERSONA_PROMPTS } from '../moa-prompts.js'
|
|
10
|
+
|
|
11
|
+
export const DEFAULT_SELF_CONSISTENCY_TEMPERATURES = Object.freeze([0.2, 0.6, 0.9])
|
|
12
|
+
export const DEFAULT_SELF_CONSISTENCY_PERSONAS = Object.freeze(['proposer', 'challenger', 'minimalist'])
|
|
13
|
+
|
|
14
|
+
/**
|
|
15
|
+
* Expands a single reference model into multiple parallel proposer slots
|
|
16
|
+
* with distinct temperatures and complementary reasoning personas.
|
|
17
|
+
*
|
|
18
|
+
* @param {Array<object>} referenceModels - Original reference model slots
|
|
19
|
+
* @param {object} [options]
|
|
20
|
+
* @param {boolean} [options.enabled=false] - Whether self-consistency expansion is enabled
|
|
21
|
+
* @param {number} [options.samples=3] - Number of candidate samples to generate (2-10)
|
|
22
|
+
* @param {Array<number>} [options.temperatures] - Array of temperatures for samples
|
|
23
|
+
* @param {Array<string>} [options.personas] - Array of persona role names
|
|
24
|
+
* @returns {{ references: Array<object>, isExpanded: boolean, sampleCount: number }}
|
|
25
|
+
*/
|
|
26
|
+
export function expandSelfConsistencySlots(referenceModels = [], options = {}) {
|
|
27
|
+
const {
|
|
28
|
+
enabled = false,
|
|
29
|
+
samples = 3,
|
|
30
|
+
temperatures = DEFAULT_SELF_CONSISTENCY_TEMPERATURES,
|
|
31
|
+
personas = DEFAULT_SELF_CONSISTENCY_PERSONAS,
|
|
32
|
+
} = options
|
|
33
|
+
|
|
34
|
+
// Expansion applies only when explicitly enabled or requested
|
|
35
|
+
if (!enabled) {
|
|
36
|
+
return {
|
|
37
|
+
references: Array.isArray(referenceModels) ? referenceModels : [],
|
|
38
|
+
isExpanded: false,
|
|
39
|
+
sampleCount: Array.isArray(referenceModels) ? referenceModels.length : 0,
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
const rawList = Array.isArray(referenceModels) && referenceModels.length > 0
|
|
44
|
+
? referenceModels
|
|
45
|
+
: [{ provider: '', model: '' }]
|
|
46
|
+
|
|
47
|
+
// Only expand when a single model slot is configured
|
|
48
|
+
if (rawList.length !== 1) {
|
|
49
|
+
return {
|
|
50
|
+
references: rawList,
|
|
51
|
+
isExpanded: false,
|
|
52
|
+
sampleCount: rawList.length,
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
const baseSlot = rawList[0] || {}
|
|
57
|
+
const count = Math.min(10, Math.max(2, Number(samples) || 3))
|
|
58
|
+
const expandedList = []
|
|
59
|
+
|
|
60
|
+
for (let i = 0; i < count; i++) {
|
|
61
|
+
const temp = Array.isArray(temperatures) && typeof temperatures[i] === 'number'
|
|
62
|
+
? temperatures[i]
|
|
63
|
+
: (DEFAULT_SELF_CONSISTENCY_TEMPERATURES[i % DEFAULT_SELF_CONSISTENCY_TEMPERATURES.length] || 0.6)
|
|
64
|
+
|
|
65
|
+
const role = Array.isArray(personas) && personas[i]
|
|
66
|
+
? personas[i]
|
|
67
|
+
: (DEFAULT_SELF_CONSISTENCY_PERSONAS[i % DEFAULT_SELF_CONSISTENCY_PERSONAS.length] || 'proposer')
|
|
68
|
+
|
|
69
|
+
expandedList.push({
|
|
70
|
+
...baseSlot,
|
|
71
|
+
label: baseSlot.label
|
|
72
|
+
? `${baseSlot.label} [T=${temp}, ${role}]`
|
|
73
|
+
: `Sample ${i + 1} (${baseSlot.model || 'model'}, T=${temp})`,
|
|
74
|
+
temperature: temp,
|
|
75
|
+
role_persona: role,
|
|
76
|
+
sampleIndex: i + 1,
|
|
77
|
+
isSelfConsistencySample: true,
|
|
78
|
+
})
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
return {
|
|
82
|
+
references: expandedList,
|
|
83
|
+
isExpanded: true,
|
|
84
|
+
sampleCount: expandedList.length,
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
/**
|
|
89
|
+
* Calculates majority agreement and self-consistency metrics across candidate outputs.
|
|
90
|
+
*
|
|
91
|
+
* @param {Array<object>} candidateOutputs - Array of successful candidate results
|
|
92
|
+
* @returns {{ agreementRatio: number, isStrongConsensus: boolean, sampleCount: number }}
|
|
93
|
+
*/
|
|
94
|
+
export function calculateSelfConsistencyAgreement(candidateOutputs = []) {
|
|
95
|
+
if (!Array.isArray(candidateOutputs) || candidateOutputs.length < 2) {
|
|
96
|
+
return {
|
|
97
|
+
agreementRatio: 1.0,
|
|
98
|
+
isStrongConsensus: true,
|
|
99
|
+
sampleCount: Array.isArray(candidateOutputs) ? candidateOutputs.length : 0,
|
|
100
|
+
}
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
const texts = candidateOutputs
|
|
104
|
+
.filter((c) => c && typeof c.text === 'string' && c.text.trim())
|
|
105
|
+
.map((c) => c.text.trim().toLowerCase().replace(/\s+/g, ' '))
|
|
106
|
+
|
|
107
|
+
if (texts.length < 2) {
|
|
108
|
+
return {
|
|
109
|
+
agreementRatio: 1.0,
|
|
110
|
+
isStrongConsensus: true,
|
|
111
|
+
sampleCount: texts.length,
|
|
112
|
+
}
|
|
113
|
+
}
|
|
114
|
+
|
|
115
|
+
// Count exact and near-match clusters
|
|
116
|
+
let maxClusterSize = 1
|
|
117
|
+
for (let i = 0; i < texts.length; i++) {
|
|
118
|
+
let clusterSize = 1
|
|
119
|
+
for (let j = 0; j < texts.length; j++) {
|
|
120
|
+
if (i === j) continue
|
|
121
|
+
// If texts are very similar (> 85% overlap)
|
|
122
|
+
if (texts[i] === texts[j] || (texts[i].length > 20 && texts[j].includes(texts[i].slice(0, 40)))) {
|
|
123
|
+
clusterSize++
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
if (clusterSize > maxClusterSize) maxClusterSize = clusterSize
|
|
127
|
+
}
|
|
128
|
+
|
|
129
|
+
const agreementRatio = Number((maxClusterSize / texts.length).toFixed(2))
|
|
130
|
+
return {
|
|
131
|
+
agreementRatio,
|
|
132
|
+
isStrongConsensus: agreementRatio >= 0.66,
|
|
133
|
+
sampleCount: texts.length,
|
|
134
|
+
}
|
|
135
|
+
}
|