@goodandready/dsh-moa 0.2.77 → 0.2.79
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -0
- package/README.ru.md +4 -0
- package/README.zh.md +4 -0
- package/lib/client.js +160 -2
- package/lib/events.js +4 -0
- package/lib/moa-schema.js +24 -0
- package/lib/pipeline/budget-guard.js +158 -0
- package/lib/pipeline/candidates.js +20 -2
- package/lib/pipeline/pareto-frontier.js +167 -0
- package/lib/pipeline/privacy-sanitizer.js +213 -0
- package/lib/pipeline/runner.js +61 -8
- package/lib/pipeline/speculative-verifier.js +212 -0
- package/lib/pricing.js +8 -3
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -56,6 +56,10 @@ Single-model AI generation often suffers from blind spots, single-perspective bi
|
|
|
56
56
|
13. **Chain-of-Verification (CoVe) Fact-Checking Gate for Code (#314)**: Independent pre-judge verification formulating factual API questions, validating modules and functions against standard library and project manifests to eliminate hallucinated methods.
|
|
57
57
|
14. **Runtime CPU & Memory Profiler Gate (#315)**: Sandboxed micro-benchmarking of candidate execution duration, ops/sec throughput, and heap usage with hard timeout protection to select algorithmically optimal code.
|
|
58
58
|
15. **Mutation Testing Gate for Candidate Tests (#316)**: Injects AST-level fault mutations to compute Mutation Score (killed vs survived mutants), weeding out tautological assertions and fake test suites.
|
|
59
|
+
16. **Hybrid Private Edge MoA & Cloud Judge Sanitizer (#306)**: Preserves confidentiality with local edge proposers on MiniPC while scrubbing absolute paths, internal IPs, and API secrets before external Cloud Judge arbitration.
|
|
60
|
+
17. **Heterogeneous Speculative Decoding (#317)**: High-throughput local edge draft coupled with single-pass cloud model verification, slashing response latency by 2-3x.
|
|
61
|
+
18. **Cost/Quality Pareto Frontier Slider (#307)**: Interactive trade-off slider mapping cost vs quality along the Pareto Frontier with real-time turn cost & latency estimation.
|
|
62
|
+
19. **Hard Token Budget Guard (#319)**: Pre-flight financial cost ceiling with dynamic auto-scaling of cloud models to local edge to guarantee turn budgets.
|
|
59
63
|
|
|
60
64
|
---
|
|
61
65
|
|
package/README.ru.md
CHANGED
|
@@ -56,6 +56,10 @@
|
|
|
56
56
|
13. **Фильтр верификации фактов и API (CoVe) (#314)**: Автономная цепочка верификации (Chain-of-Verification), формулирующая проверочные вопросы по вызовам методов и пакетам, исключающая галлюцинированные и несуществующие API библиотек.
|
|
57
57
|
14. **Профайлер производительности CPU и памяти (#315)**: Изолированный бенчмаркинг времени выполнения кандидатов, пропускной способности (ops/sec) и потребления heap-памяти с защитой от зависаний для выбора алгоритмически оптимального кода.
|
|
58
58
|
15. **Мутационное тестирование тестов кандидатов (#316)**: Внедрение AST-мутаций кода для расчёта Mutation Score (убитые vs выжившие мутанты), устраняющее ложноположительные и тавтологические тесты.
|
|
59
|
+
16. **Гибридный приватный Edge MoA (#306)**: Локальные генераторы на MiniPC сохраняют секреты и кодовую базу в контуре, маскируя пути, IP и токены перед отправкой судье в облако.
|
|
60
|
+
17. **Гетерогенное спекулятивное декодирование (#317)**: Быстрый черновик от локальной edge-модели с пакетной валидацией и доработкой флагманом в облаке с ускорением в 2-3 раза.
|
|
61
|
+
18. **Слайдер фронтира Парето (Стоимость vs Качество) (#307)**: Интерактивный регулятор баланса затрат и качества консенсуса с реактивным прогнозом цены и задержки.
|
|
62
|
+
19. **Защитный барьер бюджета токенов (#319)**: Строгий финансовый лимит стоимости сессии с динамическим авто-масштабированием и заменой облачных слотов на локальные.
|
|
59
63
|
|
|
60
64
|
---
|
|
61
65
|
|
package/README.zh.md
CHANGED
|
@@ -56,6 +56,10 @@
|
|
|
56
56
|
13. **代码事实与 API 验证链关卡 (CoVe) (#314)**:独立的裁判前事实核验,生成针对性 API 问题并对照标准库及项目配置清单,杜绝调用虚构不存在的幻觉方法。
|
|
57
57
|
14. **运行期 CPU 与内存性能分析关卡 (#315)**:在隔离沙箱中微基准测试候选方案执行延迟、吞吐量(ops/sec)与堆内存占用,带有硬超时保护,指引裁判挑选算法最优代码。
|
|
58
58
|
15. **候选测试套件变异测试关卡 (#316)**:注入 AST 级别故障变异以计算变异得分(消灭 vs 存活变异体),剔除恒真废测试与虚假测试套件。
|
|
59
|
+
16. **混合隐私边缘 MoA 与云端裁判脱敏 (#306)**:本地 MiniPC 边缘模型在内网读取代码生成方案,在提交给外部云端裁判前自动脱敏绝对路径、内网 IP 与 API 密钥。
|
|
60
|
+
17. **异构投机解码加速 (#317)**:本地小模型快速生成投机代码草稿块,云端旗舰模型单次批处理验证与完善,首包延迟降低 2-3 倍。
|
|
61
|
+
18. **成本与质量帕累托前沿滑块 (#307)**:互动式调节成本、质量与延迟的帕累托平衡,实时预估单轮开销与耗时。
|
|
62
|
+
19. **刚性 Token 预算保护与动态缩放 (#319)**:发送前严守费用上限,超出时自动将昂贵云端模型降级为本地边缘模型以确保不超支。
|
|
59
63
|
|
|
60
64
|
---
|
|
61
65
|
|
package/lib/client.js
CHANGED
|
@@ -236,7 +236,20 @@ window.__ModuleLoader__.load({
|
|
|
236
236
|
"aggregator.profiler_gate_title": "\u26A1 Automated Runtime Memory & CPU Profiler Gate",
|
|
237
237
|
"aggregator.profiler_gate_desc": "Runs isolated micro-benchmarks measuring execution latency, throughput (ops/sec), and heap memory usage.",
|
|
238
238
|
"aggregator.mutation_gate_title": "\u{1F9EC} Runtime Mutation Testing Gate for Candidate Tests",
|
|
239
|
-
"aggregator.mutation_gate_desc": "Injects synthetic faults into candidate code to test test-suite robustness, eliminating tautological and weak tests."
|
|
239
|
+
"aggregator.mutation_gate_desc": "Injects synthetic faults into candidate code to test test-suite robustness, eliminating tautological and weak tests.",
|
|
240
|
+
"aggregator.privacy_gate_title": "\u{1F6E1}\uFE0F Hybrid Private Edge MoA (Local Edge Proposers + Cloud Judge)",
|
|
241
|
+
"aggregator.privacy_gate_desc": "Scans and redacts proprietary secrets, local paths, and private network IPs before forwarding proposals to external cloud judges.",
|
|
242
|
+
"aggregator.speculative_gate_title": "\u26A1 Heterogeneous Speculative Decoding (Local Draft + Cloud Verifier)",
|
|
243
|
+
"aggregator.speculative_gate_desc": "Lightweight local edge model generates high-speed token drafts verified and completed in one fast pass by the cloud model.",
|
|
244
|
+
"aggregator.pareto_slider_title": "\u{1F4C8} Cost vs Quality Pareto Frontier Slider",
|
|
245
|
+
"aggregator.pareto_slider_desc": "Reactive balance from 100% free local edge models to ultra-rigorous flagship multi-judge consensus.",
|
|
246
|
+
"aggregator.pareto_pos_label": "Frontier Position:",
|
|
247
|
+
"aggregator.hard_budget_title": "\u{1F6E1}\uFE0F Hard Token Budget Guard with Auto-Scaling",
|
|
248
|
+
"aggregator.hard_budget_desc": "Pre-calculates turn costs against a hard financial ceiling, automatically scaling down or halting execution.",
|
|
249
|
+
"aggregator.hard_budget_limit_label": "Max Cost Per Turn (USD):",
|
|
250
|
+
"aggregator.hard_budget_policy_label": "Exceed Policy:",
|
|
251
|
+
"aggregator.hard_budget_policy_downgrade": "Auto-Downgrade to Local/Economic",
|
|
252
|
+
"aggregator.hard_budget_policy_fail": "Fail-Closed (Halt Execution)"
|
|
240
253
|
};
|
|
241
254
|
const zh = {
|
|
242
255
|
title: "\u667A\u80FD\u4F53\u6DF7\u5408 (MoA)",
|
|
@@ -463,7 +476,20 @@ window.__ModuleLoader__.load({
|
|
|
463
476
|
"aggregator.profiler_gate_title": "\u26A1 \u81EA\u52A8\u8FD0\u884C\u65F6 CPU \u4E0E\u5185\u5B58\u6027\u80FD\u5206\u6790\u5173\u5361",
|
|
464
477
|
"aggregator.profiler_gate_desc": "\u5728\u9694\u79BB\u6C99\u7BB1\u4E2D\u8FD0\u884C\u5FAE\u57FA\u51C6\u6D4B\u8BD5\uFF0C\u7CBE\u786E\u5EA6\u91CF\u6267\u884C\u5EF6\u8FDF\u3001\u541E\u5410\u91CF (ops/sec) \u548C\u5806\u5185\u5B58\u5206\u914D\u3002",
|
|
465
478
|
"aggregator.mutation_gate_title": "\u{1F9EC} \u5019\u9009\u6D4B\u8BD5\u5957\u4EF6\u8FD0\u884C\u65F6\u53D8\u5F02\u6D4B\u8BD5\u5173\u5361",
|
|
466
|
-
"aggregator.mutation_gate_desc": "\u5411\u4EE3\u7801\u6CE8\u5165\u5408\u6210\u6545\u969C\u5E76\u9A8C\u8BC1\u6D4B\u8BD5\u7684\u6355\u83B7\u80FD\u529B\uFF0C\u5254\u9664\u540C\u4E49\u53CD\u590D\u4E0E\u65E0\u6548\u7684\u865A\u5047\u6D4B\u8BD5\u3002"
|
|
479
|
+
"aggregator.mutation_gate_desc": "\u5411\u4EE3\u7801\u6CE8\u5165\u5408\u6210\u6545\u969C\u5E76\u9A8C\u8BC1\u6D4B\u8BD5\u7684\u6355\u83B7\u80FD\u529B\uFF0C\u5254\u9664\u540C\u4E49\u53CD\u590D\u4E0E\u65E0\u6548\u7684\u865A\u5047\u6D4B\u8BD5\u3002",
|
|
480
|
+
"aggregator.privacy_gate_title": "\u{1F6E1}\uFE0F \u6DF7\u5408\u9690\u79C1\u8FB9\u7F18 MoA (\u672C\u5730\u751F\u6210 + \u4E91\u7AEF\u88C1\u5224)",
|
|
481
|
+
"aggregator.privacy_gate_desc": "\u5728\u5C06\u5019\u9009\u63D0\u6848\u53D1\u9001\u7ED9\u5916\u90E8\u4E91\u7AEF\u88C1\u5224\u4E4B\u524D\uFF0C\u626B\u63CF\u5E76\u8131\u654F\u4E13\u6709\u5BC6\u94A5\u3001\u672C\u5730\u7EDD\u5BF9\u8DEF\u5F84\u4E0E\u79C1\u6709\u7F51\u7EDC IP\u3002",
|
|
482
|
+
"aggregator.speculative_gate_title": "\u26A1 \u5F02\u6784\u6295\u673A\u89E3\u7801 (\u672C\u5730\u8349\u7A3F + \u4E91\u7AEF\u9A8C\u8BC1)",
|
|
483
|
+
"aggregator.speculative_gate_desc": "\u8F7B\u91CF\u7EA7\u672C\u5730\u8FB9\u7F18\u6A21\u578B\u751F\u6210\u9AD8\u901F\u8349\u7A3F\uFF0C\u7531\u5F3A\u5927\u7684\u4E91\u7AEF\u6A21\u578B\u4E00\u6B21\u6027\u5FEB\u901F\u9A8C\u8BC1\u5E76\u5B8C\u5584\u3002",
|
|
484
|
+
"aggregator.pareto_slider_title": "\u{1F4C8} \u6210\u672C\u4E0E\u8D28\u91CF\u5E15\u7D2F\u6258\u524D\u6CBF\u6ED1\u5757 (Pareto Frontier)",
|
|
485
|
+
"aggregator.pareto_slider_desc": "\u4ECE 100% \u514D\u8D39\u672C\u5730\u8FB9\u7F18\u6A21\u578B\u5230\u8D85\u4E25\u8C28\u65D7\u8230\u591A\u88C1\u5224\u5171\u8BC6\u7684\u54CD\u5E94\u5F0F\u5E73\u8861\u8C03\u8282\u3002",
|
|
486
|
+
"aggregator.pareto_pos_label": "\u524D\u6CBF\u5E73\u8861\u4F4D\u7F6E:",
|
|
487
|
+
"aggregator.hard_budget_title": "\u{1F6E1}\uFE0F \u521A\u6027 Token \u9884\u7B97\u4FDD\u62A4\u4E0E\u52A8\u6001\u964D\u7EA7\u7F29\u653E",
|
|
488
|
+
"aggregator.hard_budget_desc": "\u5728\u53D1\u9001\u524D\u4E25\u683C\u6838\u7B97\u6574\u8F6E\u9884\u6D4B\u8D39\u7528\uFF0C\u81EA\u52A8\u964D\u7EA7\u81F3\u7ECF\u6D4E/\u672C\u5730\u6A21\u578B\u6216\u963B\u6B62\u8D85\u51FA\u9884\u7B97\u3002",
|
|
489
|
+
"aggregator.hard_budget_limit_label": "\u5355\u8F6E\u9884\u7B97\u4E0A\u9650 (USD):",
|
|
490
|
+
"aggregator.hard_budget_policy_label": "\u8D85\u989D\u7B56\u7565:",
|
|
491
|
+
"aggregator.hard_budget_policy_downgrade": "\u81EA\u52A8\u964D\u7EA7\u81F3\u672C\u5730/\u7ECF\u6D4E\u6A21\u578B",
|
|
492
|
+
"aggregator.hard_budget_policy_fail": "\u5B89\u5168\u7194\u65AD (\u505C\u6B62\u6267\u884C)"
|
|
467
493
|
};
|
|
468
494
|
function makeT(dict, fallback) {
|
|
469
495
|
return function t(key, vars) {
|
|
@@ -2710,6 +2736,138 @@ window.__ModuleLoader__.load({
|
|
|
2710
2736
|
t("aggregator.mutation_gate_desc") || "Injects synthetic faults into candidate code to test test-suite robustness, eliminating tautological and weak tests."
|
|
2711
2737
|
)
|
|
2712
2738
|
),
|
|
2739
|
+
/* Hybrid Edge Privacy Gate (#306) */
|
|
2740
|
+
React.createElement(
|
|
2741
|
+
"div",
|
|
2742
|
+
{ style: { display: "flex", flexDirection: "column", gap: 4 } },
|
|
2743
|
+
React.createElement(
|
|
2744
|
+
"label",
|
|
2745
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, fontSize: 13, fontWeight: 600, color: "var(--dsw-alias-label-primary)", cursor: "pointer" } },
|
|
2746
|
+
React.createElement("input", {
|
|
2747
|
+
type: "checkbox",
|
|
2748
|
+
checked: Boolean(currentPreset.hybrid_edge_privacy_enabled),
|
|
2749
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, hybrid_edge_privacy_enabled: e.target.checked })),
|
|
2750
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2751
|
+
}),
|
|
2752
|
+
t("aggregator.privacy_gate_title") || "\u{1F6E1}\uFE0F Hybrid Private Edge MoA (Local Edge Proposers + Cloud Judge)"
|
|
2753
|
+
),
|
|
2754
|
+
React.createElement(
|
|
2755
|
+
"div",
|
|
2756
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2757
|
+
t("aggregator.privacy_gate_desc") || "Scans and redacts proprietary secrets, local paths, and private network IPs before forwarding proposals to external cloud judges."
|
|
2758
|
+
)
|
|
2759
|
+
),
|
|
2760
|
+
/* Heterogeneous Speculative Decoding (#317) */
|
|
2761
|
+
React.createElement(
|
|
2762
|
+
"div",
|
|
2763
|
+
{ style: { display: "flex", flexDirection: "column", gap: 4 } },
|
|
2764
|
+
React.createElement(
|
|
2765
|
+
"label",
|
|
2766
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, fontSize: 13, fontWeight: 600, color: "var(--dsw-alias-label-primary)", cursor: "pointer" } },
|
|
2767
|
+
React.createElement("input", {
|
|
2768
|
+
type: "checkbox",
|
|
2769
|
+
checked: Boolean(currentPreset.speculative_draft_enabled),
|
|
2770
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, speculative_draft_enabled: e.target.checked })),
|
|
2771
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2772
|
+
}),
|
|
2773
|
+
t("aggregator.speculative_gate_title") || "\u26A1 Heterogeneous Speculative Decoding (Local Draft + Cloud Verifier)"
|
|
2774
|
+
),
|
|
2775
|
+
React.createElement(
|
|
2776
|
+
"div",
|
|
2777
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2778
|
+
t("aggregator.speculative_gate_desc") || "Lightweight local edge model generates high-speed token drafts verified and completed in one fast pass by the cloud model."
|
|
2779
|
+
)
|
|
2780
|
+
),
|
|
2781
|
+
/* Pareto Frontier Slider (#307) */
|
|
2782
|
+
React.createElement(
|
|
2783
|
+
"div",
|
|
2784
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6 } },
|
|
2785
|
+
React.createElement(
|
|
2786
|
+
"label",
|
|
2787
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, fontSize: 13, fontWeight: 600, color: "var(--dsw-alias-label-primary)", cursor: "pointer" } },
|
|
2788
|
+
React.createElement("input", {
|
|
2789
|
+
type: "checkbox",
|
|
2790
|
+
checked: Boolean(currentPreset.pareto_slider_enabled),
|
|
2791
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, pareto_slider_enabled: e.target.checked })),
|
|
2792
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2793
|
+
}),
|
|
2794
|
+
t("aggregator.pareto_slider_title") || "\u{1F4C8} Cost vs Quality Pareto Frontier Slider"
|
|
2795
|
+
),
|
|
2796
|
+
React.createElement(
|
|
2797
|
+
"div",
|
|
2798
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2799
|
+
t("aggregator.pareto_slider_desc") || "Reactive balance from 100% free local edge models to ultra-rigorous flagship multi-judge consensus."
|
|
2800
|
+
),
|
|
2801
|
+
currentPreset.pareto_slider_enabled && React.createElement(
|
|
2802
|
+
"div",
|
|
2803
|
+
{ style: { marginLeft: 22, marginTop: 4, display: "flex", alignItems: "center", gap: 10 } },
|
|
2804
|
+
React.createElement("span", { style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)" } }, t("aggregator.pareto_pos_label") || "Frontier Position:"),
|
|
2805
|
+
React.createElement("input", {
|
|
2806
|
+
type: "range",
|
|
2807
|
+
min: 0,
|
|
2808
|
+
max: 100,
|
|
2809
|
+
step: 5,
|
|
2810
|
+
value: currentPreset.pareto_slider_position ?? 50,
|
|
2811
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, pareto_slider_position: Number(e.target.value) })),
|
|
2812
|
+
style: { cursor: "pointer", flex: 1, maxWidth: 200 }
|
|
2813
|
+
}),
|
|
2814
|
+
React.createElement("span", { style: { fontSize: 12, fontWeight: 600, color: "var(--dsw-alias-label-primary)" } }, String(currentPreset.pareto_slider_position ?? 50) + "%")
|
|
2815
|
+
)
|
|
2816
|
+
),
|
|
2817
|
+
/* Hard Token Budget Guard (#319) */
|
|
2818
|
+
React.createElement(
|
|
2819
|
+
"div",
|
|
2820
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6 } },
|
|
2821
|
+
React.createElement(
|
|
2822
|
+
"label",
|
|
2823
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, fontSize: 13, fontWeight: 600, color: "var(--dsw-alias-label-primary)", cursor: "pointer" } },
|
|
2824
|
+
React.createElement("input", {
|
|
2825
|
+
type: "checkbox",
|
|
2826
|
+
checked: Boolean(currentPreset.hard_budget_guard_enabled),
|
|
2827
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, hard_budget_guard_enabled: e.target.checked })),
|
|
2828
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2829
|
+
}),
|
|
2830
|
+
t("aggregator.hard_budget_title") || "\u{1F6E1}\uFE0F Hard Token Budget Guard with Auto-Scaling"
|
|
2831
|
+
),
|
|
2832
|
+
React.createElement(
|
|
2833
|
+
"div",
|
|
2834
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2835
|
+
t("aggregator.hard_budget_desc") || "Pre-calculates turn costs against a hard financial ceiling, automatically scaling down or halting execution."
|
|
2836
|
+
),
|
|
2837
|
+
currentPreset.hard_budget_guard_enabled && React.createElement(
|
|
2838
|
+
"div",
|
|
2839
|
+
{ style: { marginLeft: 22, marginTop: 4, display: "flex", flexWrap: "wrap", alignItems: "center", gap: 12 } },
|
|
2840
|
+
React.createElement(
|
|
2841
|
+
"div",
|
|
2842
|
+
{ style: { display: "flex", alignItems: "center", gap: 6 } },
|
|
2843
|
+
React.createElement("span", { style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)" } }, t("aggregator.hard_budget_limit_label") || "Max Cost (USD):"),
|
|
2844
|
+
React.createElement("input", {
|
|
2845
|
+
type: "number",
|
|
2846
|
+
min: 1e-3,
|
|
2847
|
+
max: 10,
|
|
2848
|
+
step: 5e-3,
|
|
2849
|
+
value: currentPreset.hard_budget_limit_usd ?? 0.02,
|
|
2850
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, hard_budget_limit_usd: Number(e.target.value) })),
|
|
2851
|
+
style: { width: 80, padding: "2px 6px", fontSize: 12, borderRadius: 4, border: "1px solid var(--dsw-alias-border-primary)" }
|
|
2852
|
+
})
|
|
2853
|
+
),
|
|
2854
|
+
React.createElement(
|
|
2855
|
+
"div",
|
|
2856
|
+
{ style: { display: "flex", alignItems: "center", gap: 6 } },
|
|
2857
|
+
React.createElement("span", { style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)" } }, t("aggregator.hard_budget_policy_label") || "Policy:"),
|
|
2858
|
+
React.createElement(
|
|
2859
|
+
"select",
|
|
2860
|
+
{
|
|
2861
|
+
value: currentPreset.hard_budget_policy || "auto_downgrade",
|
|
2862
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, hard_budget_policy: e.target.value })),
|
|
2863
|
+
style: { padding: "2px 6px", fontSize: 12, borderRadius: 4, border: "1px solid var(--dsw-alias-border-primary)" }
|
|
2864
|
+
},
|
|
2865
|
+
React.createElement("option", { value: "auto_downgrade" }, t("aggregator.hard_budget_policy_downgrade") || "Auto-Downgrade"),
|
|
2866
|
+
React.createElement("option", { value: "fail_closed" }, t("aggregator.hard_budget_policy_fail") || "Fail-Closed")
|
|
2867
|
+
)
|
|
2868
|
+
)
|
|
2869
|
+
)
|
|
2870
|
+
),
|
|
2713
2871
|
/* Fallback Judges Chain */
|
|
2714
2872
|
React.createElement(
|
|
2715
2873
|
"div",
|
package/lib/events.js
CHANGED
|
@@ -15,6 +15,10 @@ export const MOA_EVENTS = {
|
|
|
15
15
|
COVE_VERIFIED: 'moa/cove-verified',
|
|
16
16
|
PROFILER_COMPLETE: 'moa/profiler-complete',
|
|
17
17
|
MUTATION_COMPLETE: 'moa/mutation-complete',
|
|
18
|
+
PRIVACY_SANITIZED: 'moa/privacy-sanitized',
|
|
19
|
+
SPECULATIVE_DRAFT: 'moa/speculative-draft',
|
|
20
|
+
PARETO_ADJUSTED: 'moa/pareto-adjusted',
|
|
21
|
+
BUDGET_SCALED: 'moa/budget-scaled',
|
|
18
22
|
CRITIQUE_COMPLETE: 'moa/critique-complete',
|
|
19
23
|
JUDGE_VERDICT: 'moa/judge-verdict',
|
|
20
24
|
CONTEXT_PRUNING: 'moa/context-pruning',
|
package/lib/moa-schema.js
CHANGED
|
@@ -100,6 +100,18 @@ export const PresetSchema = z.object({
|
|
|
100
100
|
profiler_timeout_ms: z.number().default(2000),
|
|
101
101
|
mutation_testing_enabled: z.boolean().default(false),
|
|
102
102
|
min_mutation_score: z.number().default(0.70),
|
|
103
|
+
hybrid_edge_privacy_enabled: z.boolean().default(false),
|
|
104
|
+
cloud_judge_sanitization_enabled: z.boolean().default(true),
|
|
105
|
+
redact_paths: z.boolean().default(true),
|
|
106
|
+
redact_secrets: z.boolean().default(true),
|
|
107
|
+
redact_ips: z.boolean().default(true),
|
|
108
|
+
speculative_draft_enabled: z.boolean().default(false),
|
|
109
|
+
speculative_block_size: z.number().default(256),
|
|
110
|
+
pareto_slider_enabled: z.boolean().default(false),
|
|
111
|
+
pareto_slider_position: z.number().default(50),
|
|
112
|
+
hard_budget_guard_enabled: z.boolean().default(false),
|
|
113
|
+
hard_budget_limit_usd: z.number().default(0.02),
|
|
114
|
+
hard_budget_policy: z.string().default('auto_downgrade'),
|
|
103
115
|
})
|
|
104
116
|
|
|
105
117
|
export const Config = z.object({
|
|
@@ -146,6 +158,18 @@ export const Config = z.object({
|
|
|
146
158
|
profiler_timeout_ms: z.number().default(2000).volatile(),
|
|
147
159
|
mutation_testing_enabled: z.boolean().default(false).volatile(),
|
|
148
160
|
min_mutation_score: z.number().default(0.70).volatile(),
|
|
161
|
+
hybrid_edge_privacy_enabled: z.boolean().default(false).volatile(),
|
|
162
|
+
cloud_judge_sanitization_enabled: z.boolean().default(true).volatile(),
|
|
163
|
+
redact_paths: z.boolean().default(true).volatile(),
|
|
164
|
+
redact_secrets: z.boolean().default(true).volatile(),
|
|
165
|
+
redact_ips: z.boolean().default(true).volatile(),
|
|
166
|
+
speculative_draft_enabled: z.boolean().default(false).volatile(),
|
|
167
|
+
speculative_block_size: z.number().default(256).volatile(),
|
|
168
|
+
pareto_slider_enabled: z.boolean().default(false).volatile(),
|
|
169
|
+
pareto_slider_position: z.number().default(50).volatile(),
|
|
170
|
+
hard_budget_guard_enabled: z.boolean().default(false).volatile(),
|
|
171
|
+
hard_budget_limit_usd: z.number().default(0.02).volatile(),
|
|
172
|
+
hard_budget_policy: z.string().default('auto_downgrade').volatile(),
|
|
149
173
|
smart_routing_model: ModelSlotSchema.default({ provider: '', model: '' }).volatile(),
|
|
150
174
|
prices: z.dict(PriceRow).default({}).volatile(),
|
|
151
175
|
presets: z.array(PresetSchema).default([
|
|
@@ -0,0 +1,158 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* DeepSeek Harness Mixture of Agents (MoA) — Hard Token Budget Guard & Auto-Scaling (#319)
|
|
3
|
+
*
|
|
4
|
+
* Enforces strict financial limits on inference costs. Pre-calculates end-to-end turn
|
|
5
|
+
* expenses (proposers + multi-judge + synthesis) against hard_budget_limit_usd before dispatching.
|
|
6
|
+
*
|
|
7
|
+
* Supports two operational policies when anticipated cost exceeds limit:
|
|
8
|
+
* - auto_downgrade: Automatically substitutes costly cloud slots with zero-cost local edge
|
|
9
|
+
* models or economic tiers, and scales down candidate slots until cost fits under the ceiling.
|
|
10
|
+
* - fail_closed: Halts execution immediately to eliminate unexpected API billing surprises.
|
|
11
|
+
*/
|
|
12
|
+
|
|
13
|
+
import { MOA_EVENTS, emitEvent } from '../events.js'
|
|
14
|
+
import { estimateCandidateRunCost, estimateDownstreamCost } from '../moa-budget.js'
|
|
15
|
+
|
|
16
|
+
/**
|
|
17
|
+
* Evaluates projected turn expenses and dynamically scales down ensemble if needed.
|
|
18
|
+
*
|
|
19
|
+
* @param {object} params
|
|
20
|
+
* @param {Array<object>} params.references
|
|
21
|
+
* @param {object} params.aggregator
|
|
22
|
+
* @param {number} params.limitUsd
|
|
23
|
+
* @param {string} [params.policy='auto_downgrade']
|
|
24
|
+
* @param {object} [params.prices={}]
|
|
25
|
+
* @param {number} [params.promptTokens=1500]
|
|
26
|
+
* @param {number} [params.expectedOutputTokens=2048]
|
|
27
|
+
* @param {object} [params.options]
|
|
28
|
+
* @returns {object} Budget evaluation result
|
|
29
|
+
*/
|
|
30
|
+
export function evaluateHardBudgetLimit({
|
|
31
|
+
references = [],
|
|
32
|
+
aggregator = {},
|
|
33
|
+
limitUsd = 0.02,
|
|
34
|
+
policy = 'auto_downgrade',
|
|
35
|
+
prices = {},
|
|
36
|
+
promptTokens = 1500,
|
|
37
|
+
expectedOutputTokens = 2048,
|
|
38
|
+
options = {},
|
|
39
|
+
}) {
|
|
40
|
+
const { ctx, runId, onProgress } = options
|
|
41
|
+
|
|
42
|
+
// Calculate baseline projected cost
|
|
43
|
+
const candCost = estimateCandidateRunCost(references, prices, promptTokens, expectedOutputTokens)
|
|
44
|
+
const downstreamCost = estimateDownstreamCost(
|
|
45
|
+
aggregator,
|
|
46
|
+
[],
|
|
47
|
+
references.length,
|
|
48
|
+
prices,
|
|
49
|
+
expectedOutputTokens,
|
|
50
|
+
promptTokens
|
|
51
|
+
)
|
|
52
|
+
const initialProjectedCost = Number((candCost.totalEstimatedUsd + downstreamCost).toFixed(5))
|
|
53
|
+
|
|
54
|
+
if (initialProjectedCost <= limitUsd) {
|
|
55
|
+
return {
|
|
56
|
+
ok: true,
|
|
57
|
+
scaled: false,
|
|
58
|
+
references,
|
|
59
|
+
aggregator,
|
|
60
|
+
projectedCostUsd: initialProjectedCost,
|
|
61
|
+
initialProjectedCost,
|
|
62
|
+
limitUsd,
|
|
63
|
+
warning: null,
|
|
64
|
+
}
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
// Cost exceeds limit
|
|
68
|
+
if (policy === 'fail_closed') {
|
|
69
|
+
const errorMsg = `[BudgetGuard] Projected turn cost ($${initialProjectedCost}) exceeds hard limit ($${limitUsd}). Execution halted.`
|
|
70
|
+
if (typeof onProgress === 'function') {
|
|
71
|
+
onProgress(`🛑 *${errorMsg}*\n`)
|
|
72
|
+
}
|
|
73
|
+
return {
|
|
74
|
+
ok: false,
|
|
75
|
+
scaled: false,
|
|
76
|
+
error: errorMsg,
|
|
77
|
+
projectedCostUsd: initialProjectedCost,
|
|
78
|
+
initialProjectedCost,
|
|
79
|
+
limitUsd,
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
// policy: auto_downgrade
|
|
84
|
+
let workingRefs = [...references]
|
|
85
|
+
let workingAggregator = { ...aggregator }
|
|
86
|
+
let scaled = false
|
|
87
|
+
|
|
88
|
+
// Step 1: Replace expensive cloud proposers with local edge models (zero cost)
|
|
89
|
+
for (let i = workingRefs.length - 1; i >= 0; i--) {
|
|
90
|
+
const currentSlot = workingRefs[i]
|
|
91
|
+
if (currentSlot.provider !== 'ollama' && currentSlot.provider !== 'local') {
|
|
92
|
+
workingRefs[i] = {
|
|
93
|
+
...currentSlot,
|
|
94
|
+
provider: 'ollama',
|
|
95
|
+
model: 'qwen2.5-coder:7b',
|
|
96
|
+
role_persona: currentSlot.role_persona || 'proposer',
|
|
97
|
+
}
|
|
98
|
+
scaled = true
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
const testCandCost = estimateCandidateRunCost(workingRefs, prices, promptTokens, expectedOutputTokens)
|
|
102
|
+
const testDownstream = estimateDownstreamCost(workingAggregator, [], workingRefs.length, prices, expectedOutputTokens, promptTokens)
|
|
103
|
+
if ((testCandCost.totalEstimatedUsd + testDownstream) <= limitUsd) {
|
|
104
|
+
break
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
// Step 2: If still exceeding, replace expensive aggregator with economical model or local
|
|
109
|
+
let currentCost = Number((estimateCandidateRunCost(workingRefs, prices, promptTokens, expectedOutputTokens).totalEstimatedUsd +
|
|
110
|
+
estimateDownstreamCost(workingAggregator, [], workingRefs.length, prices, expectedOutputTokens, promptTokens)).toFixed(5))
|
|
111
|
+
|
|
112
|
+
if (currentCost > limitUsd && workingAggregator.provider !== 'ollama') {
|
|
113
|
+
workingAggregator = { provider: 'deepseek', model: 'deepseek-chat' }
|
|
114
|
+
scaled = true
|
|
115
|
+
currentCost = Number((estimateCandidateRunCost(workingRefs, prices, promptTokens, expectedOutputTokens).totalEstimatedUsd +
|
|
116
|
+
estimateDownstreamCost(workingAggregator, [], workingRefs.length, prices, expectedOutputTokens, promptTokens)).toFixed(5))
|
|
117
|
+
|
|
118
|
+
if (currentCost > limitUsd) {
|
|
119
|
+
workingAggregator = { provider: 'ollama', model: 'qwen2.5-coder:7b' }
|
|
120
|
+
currentCost = 0.0
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
// Step 3: If still exceeding, prune candidates down to 1
|
|
125
|
+
if (currentCost > limitUsd && workingRefs.length > 1) {
|
|
126
|
+
workingRefs = [workingRefs[0]]
|
|
127
|
+
scaled = true
|
|
128
|
+
currentCost = Number((estimateCandidateRunCost(workingRefs, prices, promptTokens, expectedOutputTokens).totalEstimatedUsd +
|
|
129
|
+
estimateDownstreamCost(workingAggregator, [], workingRefs.length, prices, expectedOutputTokens, promptTokens)).toFixed(5))
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
if (scaled) {
|
|
133
|
+
if (typeof onProgress === 'function') {
|
|
134
|
+
onProgress(`🛡️ *Budget Guard Auto-Scaling*: Scaled ensemble from $${initialProjectedCost} to $${currentCost} to fit under $${limitUsd} ceiling.\n`)
|
|
135
|
+
}
|
|
136
|
+
|
|
137
|
+
if (ctx) {
|
|
138
|
+
emitEvent(ctx, MOA_EVENTS.BUDGET_SCALED, {
|
|
139
|
+
runId,
|
|
140
|
+
originalCostUsd: initialProjectedCost,
|
|
141
|
+
scaledCostUsd: currentCost,
|
|
142
|
+
limitUsd,
|
|
143
|
+
policy,
|
|
144
|
+
})
|
|
145
|
+
}
|
|
146
|
+
}
|
|
147
|
+
|
|
148
|
+
return {
|
|
149
|
+
ok: true,
|
|
150
|
+
scaled,
|
|
151
|
+
references: workingRefs,
|
|
152
|
+
aggregator: workingAggregator,
|
|
153
|
+
projectedCostUsd: currentCost,
|
|
154
|
+
initialProjectedCost,
|
|
155
|
+
limitUsd,
|
|
156
|
+
warning: scaled ? `Budget Guard Auto-Scaled: initial $${initialProjectedCost} $\\to$ $${currentCost} (ceiling $${limitUsd})` : null,
|
|
157
|
+
}
|
|
158
|
+
}
|
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { createSpeculativeCallLlm } from './speculative-verifier.js'
|
|
1
2
|
import { createCachingCallLlm } from './semantic-cache.js'
|
|
2
3
|
import { runSpeculativeMoa } from '../moa-speculative.js'
|
|
3
4
|
import { runAdversarialTddLoop } from '../moa-adversarial-tdd.js'
|
|
@@ -47,6 +48,23 @@ export async function stageCandidateRound1({
|
|
|
47
48
|
ctx,
|
|
48
49
|
})
|
|
49
50
|
: callLlm
|
|
51
|
+
let candidateCallLlm = effectiveCallLlm
|
|
52
|
+
const draftModelSlot = typeof preset?.speculative_draft_model === 'object' && preset?.speculative_draft_model?.model
|
|
53
|
+
? preset.speculative_draft_model
|
|
54
|
+
: typeof preset?.speculative_draft_model === 'string' && preset?.speculative_draft_model
|
|
55
|
+
? { provider: 'ollama', model: preset.speculative_draft_model }
|
|
56
|
+
: null
|
|
57
|
+
|
|
58
|
+
if (preset?.speculative_draft_enabled && draftModelSlot) {
|
|
59
|
+
candidateCallLlm = createSpeculativeCallLlm(effectiveCallLlm, {
|
|
60
|
+
draftSlot: draftModelSlot,
|
|
61
|
+
blockSize: preset.speculative_block_size || 256,
|
|
62
|
+
ctx,
|
|
63
|
+
runId,
|
|
64
|
+
onProgress,
|
|
65
|
+
})
|
|
66
|
+
}
|
|
67
|
+
|
|
50
68
|
const lastMsgText = enrichedMessages[enrichedMessages.length - 1]?.content || ''
|
|
51
69
|
const adaptiveRefs = preset?.adaptive_personas_enabled
|
|
52
70
|
? assignAdaptivePersonas(effectiveReferences, typeof lastMsgText === 'string' ? lastMsgText : '').references
|
|
@@ -75,7 +93,7 @@ export async function stageCandidateRound1({
|
|
|
75
93
|
|
|
76
94
|
if (!referenceOutputs) {
|
|
77
95
|
referenceOutputs = preset?.subagent_swarm_enabled
|
|
78
|
-
? await runSubagentSwarm(adaptiveRefs, enrichedMessages, { temperature: refTemp, maxTokens, signal, cwd, runId },
|
|
96
|
+
? await runSubagentSwarm(adaptiveRefs, enrichedMessages, { temperature: refTemp, maxTokens, signal, cwd, runId }, candidateCallLlm, onProgress)
|
|
79
97
|
: await runReferencesParallel(
|
|
80
98
|
adaptiveRefs,
|
|
81
99
|
enrichedMessages,
|
|
@@ -93,7 +111,7 @@ export async function stageCandidateRound1({
|
|
|
93
111
|
maxRetries: candidateRetries,
|
|
94
112
|
signal,
|
|
95
113
|
},
|
|
96
|
-
|
|
114
|
+
candidateCallLlm,
|
|
97
115
|
onProgress
|
|
98
116
|
)
|
|
99
117
|
}
|
|
@@ -0,0 +1,167 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* DeepSeek Harness Mixture of Agents (MoA) — Cost/Quality Pareto Frontier Slider (#307)
|
|
3
|
+
*
|
|
4
|
+
* Provides reactive calculation of the optimal ensemble trade-off between Cost ($),
|
|
5
|
+
* Quality (Rigorous Consensus), and Latency (seconds) along the Pareto Frontier.
|
|
6
|
+
* Dynamically maps a single slider value (0–100%) to curated model configurations:
|
|
7
|
+
* - 0–25%: Local Free Edge Tier ($0.00 cost, ultra-low latency)
|
|
8
|
+
* - 26–50%: Economic Tier (Low-cost cloud models + deepseek-chat)
|
|
9
|
+
* - 51–75%: Balanced Production Tier (Diverse candidates + deepseek-reasoner / Sonnet)
|
|
10
|
+
* - 76–100%: Ultra-Rigorous Consensus Tier (Flagship ensemble + Multi-Judge panel)
|
|
11
|
+
*/
|
|
12
|
+
|
|
13
|
+
import { MOA_EVENTS, emitEvent } from '../events.js'
|
|
14
|
+
import { estimateCandidateRunCost, estimateDownstreamCost } from '../moa-budget.js'
|
|
15
|
+
|
|
16
|
+
export const PARETO_TIERS = [
|
|
17
|
+
{
|
|
18
|
+
maxPos: 25,
|
|
19
|
+
name: 'local_free',
|
|
20
|
+
label: '⚡ 100% Free / Local Edge',
|
|
21
|
+
description: 'Zero API costs. Local models on MiniPC (Qwen 2.5 Coder / Ollama) for maximum speed and privacy.',
|
|
22
|
+
defaultReferences: [
|
|
23
|
+
{ provider: 'ollama', model: 'qwen2.5-coder:7b', role_persona: 'proposer' },
|
|
24
|
+
{ provider: 'ollama', model: 'deepseek-r1:8b', role_persona: 'challenger' },
|
|
25
|
+
],
|
|
26
|
+
defaultAggregator: { provider: 'ollama', model: 'qwen2.5-coder:7b' },
|
|
27
|
+
estimatedCostUsd: 0.0,
|
|
28
|
+
estimatedLatencySec: 4,
|
|
29
|
+
multiJudge: false,
|
|
30
|
+
},
|
|
31
|
+
{
|
|
32
|
+
maxPos: 50,
|
|
33
|
+
name: 'economic',
|
|
34
|
+
label: '💡 Economic Tier',
|
|
35
|
+
description: 'Low-cost fast models. Local edge proposers verified by an economical cloud aggregator.',
|
|
36
|
+
defaultReferences: [
|
|
37
|
+
{ provider: 'ollama', model: 'qwen2.5-coder:7b', role_persona: 'proposer' },
|
|
38
|
+
{ provider: 'deepseek', model: 'deepseek-chat', role_persona: 'challenger' },
|
|
39
|
+
],
|
|
40
|
+
defaultAggregator: { provider: 'deepseek', model: 'deepseek-chat' },
|
|
41
|
+
estimatedCostUsd: 0.002,
|
|
42
|
+
estimatedLatencySec: 7,
|
|
43
|
+
multiJudge: false,
|
|
44
|
+
},
|
|
45
|
+
{
|
|
46
|
+
maxPos: 75,
|
|
47
|
+
name: 'balanced',
|
|
48
|
+
label: '⚖️ Optimal Balanced Tier',
|
|
49
|
+
description: 'High hypothesis diversity with reasoning models and flagship aggregator.',
|
|
50
|
+
defaultReferences: [
|
|
51
|
+
{ provider: 'deepseek', model: 'deepseek-chat', role_persona: 'proposer' },
|
|
52
|
+
{ provider: 'deepseek', model: 'deepseek-reasoner', role_persona: 'challenger' },
|
|
53
|
+
{ provider: 'ollama', model: 'qwen2.5-coder:7b', role_persona: 'minimalist' },
|
|
54
|
+
],
|
|
55
|
+
defaultAggregator: { provider: 'deepseek', model: 'deepseek-reasoner' },
|
|
56
|
+
estimatedCostUsd: 0.009,
|
|
57
|
+
estimatedLatencySec: 12,
|
|
58
|
+
multiJudge: false,
|
|
59
|
+
},
|
|
60
|
+
{
|
|
61
|
+
maxPos: 100,
|
|
62
|
+
name: 'ultra_rigorous',
|
|
63
|
+
label: '🏆 Ultra-Rigorous Consensus',
|
|
64
|
+
description: 'Maximum accuracy. Flagship reasoning models with multi-judge panel and adversarial peer critique.',
|
|
65
|
+
defaultReferences: [
|
|
66
|
+
{ provider: 'deepseek', model: 'deepseek-reasoner', role_persona: 'proposer' },
|
|
67
|
+
{ provider: 'anthropic', model: 'claude-3-5-sonnet', role_persona: 'challenger' },
|
|
68
|
+
{ provider: 'openai', model: 'gpt-4o', role_persona: 'adversary' },
|
|
69
|
+
],
|
|
70
|
+
defaultAggregator: { provider: 'deepseek', model: 'deepseek-reasoner' },
|
|
71
|
+
estimatedCostUsd: 0.038,
|
|
72
|
+
estimatedLatencySec: 22,
|
|
73
|
+
multiJudge: true,
|
|
74
|
+
},
|
|
75
|
+
]
|
|
76
|
+
|
|
77
|
+
/**
|
|
78
|
+
* Returns the Pareto tier definition for a given slider position (0-100).
|
|
79
|
+
*
|
|
80
|
+
* @param {number} position
|
|
81
|
+
* @returns {object} Tier configuration
|
|
82
|
+
*/
|
|
83
|
+
export function getParetoProfile(position = 50) {
|
|
84
|
+
const clamped = Math.max(0, Math.min(100, Number(position) || 0))
|
|
85
|
+
for (const tier of PARETO_TIERS) {
|
|
86
|
+
if (clamped <= tier.maxPos) {
|
|
87
|
+
return { ...tier, position: clamped }
|
|
88
|
+
}
|
|
89
|
+
}
|
|
90
|
+
return { ...PARETO_TIERS[PARETO_TIERS.length - 1], position: clamped }
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
/**
|
|
94
|
+
* Calculates dynamically predicted cost and latency for an ensemble along the Pareto Frontier.
|
|
95
|
+
*
|
|
96
|
+
* @param {number} sliderPos
|
|
97
|
+
* @param {Array<object>} [availableModels]
|
|
98
|
+
* @param {object} [options]
|
|
99
|
+
* @returns {object} Ensemble recommendation
|
|
100
|
+
*/
|
|
101
|
+
export function calculateParetoEnsemble(sliderPos = 50, availableModels = [], options = {}) {
|
|
102
|
+
const profile = getParetoProfile(sliderPos)
|
|
103
|
+
const { prices = {}, promptTokens = 1500, expectedOutputTokens = 2048 } = options
|
|
104
|
+
|
|
105
|
+
// Calculate upfront cost
|
|
106
|
+
const candCost = estimateCandidateRunCost(profile.defaultReferences, prices, promptTokens, expectedOutputTokens)
|
|
107
|
+
const downstreamCost = estimateDownstreamCost(
|
|
108
|
+
profile.defaultAggregator,
|
|
109
|
+
profile.multiJudge ? profile.defaultReferences : [],
|
|
110
|
+
profile.defaultReferences.length,
|
|
111
|
+
prices,
|
|
112
|
+
expectedOutputTokens,
|
|
113
|
+
promptTokens
|
|
114
|
+
)
|
|
115
|
+
|
|
116
|
+
const calculatedCostUsd = profile.name === 'local_free'
|
|
117
|
+
? 0.0
|
|
118
|
+
: Number((candCost.totalEstimatedUsd + downstreamCost).toFixed(5))
|
|
119
|
+
|
|
120
|
+
return {
|
|
121
|
+
tier: profile.name,
|
|
122
|
+
label: profile.label,
|
|
123
|
+
description: profile.description,
|
|
124
|
+
sliderPosition: profile.position,
|
|
125
|
+
estimatedCostUsd: calculatedCostUsd > 0 ? calculatedCostUsd : profile.estimatedCostUsd,
|
|
126
|
+
estimatedLatencySec: profile.estimatedLatencySec,
|
|
127
|
+
recommendedReferences: profile.defaultReferences,
|
|
128
|
+
recommendedAggregator: profile.defaultAggregator,
|
|
129
|
+
multiJudgeEnabled: profile.multiJudge,
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
/**
|
|
134
|
+
* Adapts preset configuration based on Pareto slider position if enabled.
|
|
135
|
+
*
|
|
136
|
+
* @param {object} preset
|
|
137
|
+
* @param {object} [options]
|
|
138
|
+
* @returns {object} Adjusted preset
|
|
139
|
+
*/
|
|
140
|
+
export function applyParetoAdjustment(preset, options = {}) {
|
|
141
|
+
if (!preset?.pareto_slider_enabled) return preset
|
|
142
|
+
|
|
143
|
+
const { ctx, runId, onProgress } = options
|
|
144
|
+
const position = preset.pareto_slider_position ?? 50
|
|
145
|
+
const recommendation = calculateParetoEnsemble(position, options.availableModels || [], options)
|
|
146
|
+
|
|
147
|
+
if (typeof onProgress === 'function') {
|
|
148
|
+
onProgress(`📈 *Pareto Frontier*: Active profile ${recommendation.label} (~$${recommendation.estimatedCostUsd}, ~${recommendation.estimatedLatencySec}s)\n`)
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
if (ctx) {
|
|
152
|
+
emitEvent(ctx, MOA_EVENTS.PARETO_ADJUSTED, {
|
|
153
|
+
runId,
|
|
154
|
+
position,
|
|
155
|
+
tier: recommendation.tier,
|
|
156
|
+
estimatedCostUsd: recommendation.estimatedCostUsd,
|
|
157
|
+
estimatedLatencySec: recommendation.estimatedLatencySec,
|
|
158
|
+
})
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
return {
|
|
162
|
+
...preset,
|
|
163
|
+
reference_models: recommendation.recommendedReferences,
|
|
164
|
+
aggregator: recommendation.recommendedAggregator,
|
|
165
|
+
multi_judge_enabled: recommendation.multiJudgeEnabled || preset.multi_judge_enabled,
|
|
166
|
+
}
|
|
167
|
+
}
|
|
@@ -0,0 +1,213 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* DeepSeek Harness Mixture of Agents (MoA) — Hybrid Private Edge & Privacy Sanitizer (#306)
|
|
3
|
+
*
|
|
4
|
+
* Protects project confidentiality in hybrid deployments by scrubbing local absolute paths,
|
|
5
|
+
* internal private IPs, API tokens, JWTs, and proprietary secrets before forwarding prompts
|
|
6
|
+
* or candidate proposals to external Cloud Judges. Preserves unsanitized originals locally
|
|
7
|
+
* for workspace promotion.
|
|
8
|
+
*/
|
|
9
|
+
|
|
10
|
+
import { MOA_EVENTS, emitEvent } from '../events.js'
|
|
11
|
+
|
|
12
|
+
/**
|
|
13
|
+
* Common regex definitions for sensitive data redaction.
|
|
14
|
+
*/
|
|
15
|
+
export const SANITIZATION_PATTERNS = {
|
|
16
|
+
OPENAI_OR_GENERIC_KEY: /sk-[a-zA-Z0-9_\-]{20,}/g,
|
|
17
|
+
BEARER_TOKEN: /Bearer\s+[a-zA-Z0-9_\-\.]{20,}/gi,
|
|
18
|
+
JWT: /ey[A-Za-z0-9_\-]{15,}\.ey[A-Za-z0-9_\-]{15,}\.[A-Za-z0-9_\-]{10,}/g,
|
|
19
|
+
AWS_KEY: /AKIA[0-9A-Z]{16}/g,
|
|
20
|
+
PRIVATE_KEY: /-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----[\s\S]*?-----END (?:RSA |EC |OPENSSH )?PRIVATE KEY-----/g,
|
|
21
|
+
SECRET_ASSIGNMENT: /(?<=(?:api[_-]?key|secret|password|token|auth)\s*[:=]\s*["'])[a-zA-Z0-9_\-\.\/+=]{8,}(?=["'])/gi,
|
|
22
|
+
UNIX_HOME_PATH: /(?:\/(?:home|Users)\/[a-zA-Z0-9_\-]+(?:\/[a-zA-Z0-9_\-\.]+)*)/g,
|
|
23
|
+
WIN_USER_PATH: /(?:[A-Za-z]:\\(?:Users|Documents and Settings)\\[a-zA-Z0-9_\-]+(?:\\[a-zA-Z0-9_\-\.]+)*)/g,
|
|
24
|
+
PRIVATE_IP: /\b(?:192\.168\.\d{1,3}\.\d{1,3}|10\.\d{1,3}\.\d{1,3}\.\d{1,3}|172\.(?:1[6-9]|2\d|3[0-1])\.\d{1,3}\.\d{1,3})\b/g,
|
|
25
|
+
EMAIL: /\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b/g,
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
/**
|
|
29
|
+
* Sanitizes a text string by redacting sensitive secrets, internal paths, and PII.
|
|
30
|
+
*
|
|
31
|
+
* @param {string} text
|
|
32
|
+
* @param {object} [options]
|
|
33
|
+
* @returns {{ sanitizedText: string, redactionCount: number, redactingEntries: Array<object> }}
|
|
34
|
+
*/
|
|
35
|
+
export function sanitizeSensitiveText(text, options = {}) {
|
|
36
|
+
if (!text || typeof text !== 'string') {
|
|
37
|
+
return { sanitizedText: text || '', redactionCount: 0, redactingEntries: [] }
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
const {
|
|
41
|
+
redactSecrets = true,
|
|
42
|
+
redactPaths = true,
|
|
43
|
+
redactIps = true,
|
|
44
|
+
redactEmails = true,
|
|
45
|
+
} = options
|
|
46
|
+
|
|
47
|
+
let result = text
|
|
48
|
+
let redactionCount = 0
|
|
49
|
+
const redactingEntries = []
|
|
50
|
+
|
|
51
|
+
// 1. Secrets & API Keys
|
|
52
|
+
if (redactSecrets) {
|
|
53
|
+
result = result.replace(SANITIZATION_PATTERNS.PRIVATE_KEY, () => {
|
|
54
|
+
redactionCount++
|
|
55
|
+
redactingEntries.push({ type: 'PRIVATE_KEY' })
|
|
56
|
+
return '[REDACTED_PRIVATE_KEY]'
|
|
57
|
+
})
|
|
58
|
+
|
|
59
|
+
result = result.replace(SANITIZATION_PATTERNS.OPENAI_OR_GENERIC_KEY, () => {
|
|
60
|
+
redactionCount++
|
|
61
|
+
redactingEntries.push({ type: 'API_KEY' })
|
|
62
|
+
return '[REDACTED_API_KEY]'
|
|
63
|
+
})
|
|
64
|
+
|
|
65
|
+
result = result.replace(SANITIZATION_PATTERNS.JWT, () => {
|
|
66
|
+
redactionCount++
|
|
67
|
+
redactingEntries.push({ type: 'JWT' })
|
|
68
|
+
return '[REDACTED_JWT]'
|
|
69
|
+
})
|
|
70
|
+
|
|
71
|
+
result = result.replace(SANITIZATION_PATTERNS.AWS_KEY, () => {
|
|
72
|
+
redactionCount++
|
|
73
|
+
redactingEntries.push({ type: 'AWS_KEY' })
|
|
74
|
+
return '[REDACTED_AWS_KEY]'
|
|
75
|
+
})
|
|
76
|
+
|
|
77
|
+
result = result.replace(SANITIZATION_PATTERNS.SECRET_ASSIGNMENT, () => {
|
|
78
|
+
redactionCount++
|
|
79
|
+
redactingEntries.push({ type: 'SECRET_VALUE' })
|
|
80
|
+
return '[REDACTED_SECRET]'
|
|
81
|
+
})
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
// 2. Private Paths
|
|
85
|
+
if (redactPaths) {
|
|
86
|
+
result = result.replace(SANITIZATION_PATTERNS.UNIX_HOME_PATH, (match) => {
|
|
87
|
+
redactionCount++
|
|
88
|
+
const parts = match.split('/')
|
|
89
|
+
const subpath = parts.slice(3).join('/')
|
|
90
|
+
return subpath ? `/workspace/${subpath}` : '/workspace'
|
|
91
|
+
})
|
|
92
|
+
|
|
93
|
+
result = result.replace(SANITIZATION_PATTERNS.WIN_USER_PATH, (match) => {
|
|
94
|
+
redactionCount++
|
|
95
|
+
const parts = match.split('\\')
|
|
96
|
+
const subpath = parts.slice(3).join('\\')
|
|
97
|
+
return subpath ? `C:\\workspace\\${subpath}` : 'C:\\workspace'
|
|
98
|
+
})
|
|
99
|
+
}
|
|
100
|
+
|
|
101
|
+
// 3. Private IP addresses
|
|
102
|
+
if (redactIps) {
|
|
103
|
+
result = result.replace(SANITIZATION_PATTERNS.PRIVATE_IP, () => {
|
|
104
|
+
redactionCount++
|
|
105
|
+
redactingEntries.push({ type: 'INTERNAL_IP' })
|
|
106
|
+
return '[REDACTED_INTERNAL_IP]'
|
|
107
|
+
})
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
// 4. Email addresses
|
|
111
|
+
if (redactEmails) {
|
|
112
|
+
result = result.replace(SANITIZATION_PATTERNS.EMAIL, () => {
|
|
113
|
+
redactionCount++
|
|
114
|
+
redactingEntries.push({ type: 'EMAIL' })
|
|
115
|
+
return '[REDACTED_EMAIL]'
|
|
116
|
+
})
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
return {
|
|
120
|
+
sanitizedText: result,
|
|
121
|
+
redactionCount,
|
|
122
|
+
redactingEntries,
|
|
123
|
+
}
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
/**
|
|
127
|
+
* Sanitizes candidate solutions before forwarding them to an external Cloud Judge.
|
|
128
|
+
* Preserves the original unmodified code in candidate.originalOutput for safe local promotion.
|
|
129
|
+
*
|
|
130
|
+
* @param {Array<object>} candidates
|
|
131
|
+
* @param {object} [options]
|
|
132
|
+
* @returns {object} Sanitization result
|
|
133
|
+
*/
|
|
134
|
+
export function sanitizeCandidatesForCloudJudge(candidates, options = {}) {
|
|
135
|
+
const {
|
|
136
|
+
ctx,
|
|
137
|
+
runId,
|
|
138
|
+
onProgress,
|
|
139
|
+
redactSecrets = true,
|
|
140
|
+
redactPaths = true,
|
|
141
|
+
redactIps = true,
|
|
142
|
+
redactEmails = true,
|
|
143
|
+
} = options
|
|
144
|
+
|
|
145
|
+
if (!Array.isArray(candidates) || candidates.length === 0) {
|
|
146
|
+
return {
|
|
147
|
+
sanitizedCandidates: [],
|
|
148
|
+
totalRedactions: 0,
|
|
149
|
+
privacyDirective: null,
|
|
150
|
+
}
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
let totalRedactions = 0
|
|
154
|
+
const sanitizedCandidates = candidates.map((cand, idx) => {
|
|
155
|
+
const rawText = typeof cand === 'string' ? cand : cand.output || cand.text || cand.code || ''
|
|
156
|
+
const { sanitizedText, redactionCount, redactingEntries } = sanitizeSensitiveText(rawText, {
|
|
157
|
+
redactSecrets,
|
|
158
|
+
redactPaths,
|
|
159
|
+
redactIps,
|
|
160
|
+
redactEmails,
|
|
161
|
+
})
|
|
162
|
+
|
|
163
|
+
totalRedactions += redactionCount
|
|
164
|
+
|
|
165
|
+
if (typeof cand === 'object' && cand !== null) {
|
|
166
|
+
if (!cand.originalOutput) {
|
|
167
|
+
cand.originalOutput = rawText
|
|
168
|
+
}
|
|
169
|
+
cand.sanitizedOutput = sanitizedText
|
|
170
|
+
cand.redactionCount = redactionCount
|
|
171
|
+
}
|
|
172
|
+
|
|
173
|
+
if (ctx && redactionCount > 0) {
|
|
174
|
+
emitEvent(ctx, MOA_EVENTS.PRIVACY_SANITIZED, {
|
|
175
|
+
runId,
|
|
176
|
+
candidateIndex: idx,
|
|
177
|
+
model: cand?.model || `Candidate ${idx + 1}`,
|
|
178
|
+
redactionCount,
|
|
179
|
+
types: [...new Set(redactingEntries.map((e) => e.type))],
|
|
180
|
+
})
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
return typeof cand === 'object' && cand !== null
|
|
184
|
+
? { ...cand, output: sanitizedText, text: sanitizedText }
|
|
185
|
+
: sanitizedText
|
|
186
|
+
})
|
|
187
|
+
|
|
188
|
+
if (typeof onProgress === 'function' && totalRedactions > 0) {
|
|
189
|
+
onProgress(`🛡️ *Hybrid Edge Privacy*: Sanitized ${totalRedactions} sensitive token(s) (secrets, paths, IPs) before cloud judge.\n`)
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
const privacyDirective = totalRedactions > 0 ? formatPrivacyDirective(totalRedactions) : null
|
|
193
|
+
|
|
194
|
+
return {
|
|
195
|
+
sanitizedCandidates,
|
|
196
|
+
totalRedactions,
|
|
197
|
+
privacyDirective,
|
|
198
|
+
}
|
|
199
|
+
}
|
|
200
|
+
|
|
201
|
+
/**
|
|
202
|
+
* Formats a privacy directive header for the Cloud Judge prompt.
|
|
203
|
+
*
|
|
204
|
+
* @param {number} totalRedactions
|
|
205
|
+
* @returns {string}
|
|
206
|
+
*/
|
|
207
|
+
export function formatPrivacyDirective(totalRedactions) {
|
|
208
|
+
return [
|
|
209
|
+
'### 🛡️ PRIVACY & PII SANITIZATION ACTIVE',
|
|
210
|
+
`Notice: ${totalRedactions} internal system path(s), private network IP(s), or secret token(s) have been sanitized with [REDACTED_*] or relative paths to preserve user confidentiality.`,
|
|
211
|
+
'Evaluate the candidate solutions based on their architecture, correctness, and logic.',
|
|
212
|
+
].join('\n')
|
|
213
|
+
}
|
package/lib/pipeline/runner.js
CHANGED
|
@@ -1,3 +1,6 @@
|
|
|
1
|
+
import { applyParetoAdjustment } from './pareto-frontier.js'
|
|
2
|
+
import { evaluateHardBudgetLimit } from './budget-guard.js'
|
|
3
|
+
import { sanitizeCandidatesForCloudJudge } from './privacy-sanitizer.js'
|
|
1
4
|
import { evaluateJudgeEscalation } from './escalation.js'
|
|
2
5
|
import { calculateSelfConsistencyAgreement } from './self-consistency.js'
|
|
3
6
|
import { pruneCandidateEnsemble, formatPruningDirective } from './context-pruner.js'
|
|
@@ -49,9 +52,36 @@ export async function runMoAPipeline({
|
|
|
49
52
|
timestamp: new Date(startTime).toISOString(),
|
|
50
53
|
})
|
|
51
54
|
|
|
55
|
+
// Pre-Stage 0.5: Pareto Frontier Slider & Hard Budget Guard (#307, #319)
|
|
56
|
+
let effectivePreset = preset
|
|
57
|
+
if (effectivePreset?.pareto_slider_enabled) {
|
|
58
|
+
effectivePreset = applyParetoAdjustment(effectivePreset, { ctx, runId, onProgress, prices })
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
if (effectivePreset?.hard_budget_guard_enabled) {
|
|
62
|
+
const budgetRes = evaluateHardBudgetLimit({
|
|
63
|
+
references: effectivePreset.reference_models || [],
|
|
64
|
+
aggregator: effectivePreset.aggregator || {},
|
|
65
|
+
limitUsd: effectivePreset.hard_budget_limit_usd ?? 0.02,
|
|
66
|
+
policy: effectivePreset.hard_budget_policy || 'auto_downgrade',
|
|
67
|
+
prices,
|
|
68
|
+
options: { ctx, runId, onProgress },
|
|
69
|
+
})
|
|
70
|
+
if (!budgetRes.ok) {
|
|
71
|
+
return { error: new Error(budgetRes.error) }
|
|
72
|
+
}
|
|
73
|
+
if (budgetRes.scaled) {
|
|
74
|
+
effectivePreset = {
|
|
75
|
+
...effectivePreset,
|
|
76
|
+
reference_models: budgetRes.references,
|
|
77
|
+
aggregator: budgetRes.aggregator,
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
|
|
52
82
|
// Stage 1: Context & Configuration
|
|
53
83
|
const ctxStage = await stageProjectContext({
|
|
54
|
-
userPrompt, messages, preset, cwd, prices, maxTokens:
|
|
84
|
+
userPrompt, messages, preset: effectivePreset, cwd, prices, maxTokens: effectivePreset?.max_tokens, startTime, onProgress, ctx,
|
|
55
85
|
})
|
|
56
86
|
if (!ctxStage.ok) return ctxStage.abortPayload
|
|
57
87
|
|
|
@@ -269,6 +299,30 @@ export async function runMoAPipeline({
|
|
|
269
299
|
mutationDirective = mutationGateResult.mutationDirective
|
|
270
300
|
}
|
|
271
301
|
|
|
302
|
+
// Pre-Stage 5.12: Hybrid Edge Privacy Sanitizer Gate (#306)
|
|
303
|
+
let privacyDirective = null
|
|
304
|
+
let candidatesForJudge = syntaxResult.validCandidates.length > 0 ? syntaxResult.validCandidates : successfulRefs
|
|
305
|
+
let sanitizedReferenceOutputs = referenceOutputs
|
|
306
|
+
if (preset?.hybrid_edge_privacy_enabled || preset?.cloud_judge_sanitization_enabled) {
|
|
307
|
+
const sanitizeResult = sanitizeCandidatesForCloudJudge(candidatesForJudge, {
|
|
308
|
+
ctx,
|
|
309
|
+
runId,
|
|
310
|
+
onProgress,
|
|
311
|
+
redactSecrets: preset.redact_secrets !== false,
|
|
312
|
+
redactPaths: preset.redact_paths !== false,
|
|
313
|
+
redactIps: preset.redact_ips !== false,
|
|
314
|
+
})
|
|
315
|
+
candidatesForJudge = sanitizeResult.sanitizedCandidates
|
|
316
|
+
privacyDirective = sanitizeResult.privacyDirective
|
|
317
|
+
sanitizedReferenceOutputs = referenceOutputs.map((r) => {
|
|
318
|
+
const matching = candidatesForJudge.find((c) => c.index === r.index || c.model === r.model || c.label === r.label)
|
|
319
|
+
if (matching && matching.sanitizedOutput) {
|
|
320
|
+
return { ...r, text: matching.sanitizedOutput, output: matching.sanitizedOutput }
|
|
321
|
+
}
|
|
322
|
+
return r
|
|
323
|
+
})
|
|
324
|
+
}
|
|
325
|
+
|
|
272
326
|
// Pre-Stage 6: Dynamic Consensus Threshold & Multi-Judge Escalation (#300)
|
|
273
327
|
const escalationResult = evaluateJudgeEscalation({
|
|
274
328
|
candidateOutputs: syntaxResult.validCandidates.length > 0 ? syntaxResult.validCandidates : successfulRefs,
|
|
@@ -282,7 +336,6 @@ export async function runMoAPipeline({
|
|
|
282
336
|
let effectivePrimaryJudge = ctxStage.primaryJudge
|
|
283
337
|
let effectiveFallbackJudges = ctxStage.fallbackJudges
|
|
284
338
|
let effectiveJudgesChain = ctxStage.judgesChain
|
|
285
|
-
let effectivePreset = preset
|
|
286
339
|
|
|
287
340
|
if (escalationResult.escalated) {
|
|
288
341
|
emitEvent(ctx, MOA_EVENTS.JUDGE_ESCALATION, {
|
|
@@ -317,9 +370,9 @@ export async function runMoAPipeline({
|
|
|
317
370
|
// Stage 6: Multi-Judge Panel
|
|
318
371
|
const mjRes = await stageMultiJudge({
|
|
319
372
|
preset: effectivePreset,
|
|
320
|
-
successfulRefs:
|
|
321
|
-
referenceOutputs, primaryJudge: effectivePrimaryJudge,
|
|
322
|
-
userPrompt, judgeCriteria: [ctxStage.judgeCriteria, coveDirective, securityDirective, profilerDirective, mutationDirective].filter(Boolean).join('\n\n'), prices, maxTokens: ctxStage.maxTokens,
|
|
373
|
+
successfulRefs: candidatesForJudge,
|
|
374
|
+
referenceOutputs: sanitizedReferenceOutputs, primaryJudge: effectivePrimaryJudge,
|
|
375
|
+
userPrompt, judgeCriteria: [ctxStage.judgeCriteria, coveDirective, securityDirective, profilerDirective, mutationDirective, privacyDirective].filter(Boolean).join('\n\n'), prices, maxTokens: ctxStage.maxTokens,
|
|
323
376
|
callLlm, signal, onProgress, cwd, runId, isRefinement: ctxStage.isRefinement,
|
|
324
377
|
isFastMode: ctxStage.isFastMode, collectedCtx: ctxStage.collectedCtx, startTime,
|
|
325
378
|
})
|
|
@@ -348,8 +401,8 @@ export async function runMoAPipeline({
|
|
|
348
401
|
: null
|
|
349
402
|
|
|
350
403
|
// Pre-Judge Stage 6.8: Cross-Candidate Unchanged Base Context Pruning (#305)
|
|
351
|
-
let synthCandidateRefs =
|
|
352
|
-
let synthReferenceOutputs =
|
|
404
|
+
let synthCandidateRefs = candidatesForJudge
|
|
405
|
+
let synthReferenceOutputs = sanitizedReferenceOutputs
|
|
353
406
|
let pruningDirective = null
|
|
354
407
|
if (ctxStage.contextPruningEnabled && ctxStage.collectedCtx?.files?.length > 0) {
|
|
355
408
|
const pruneRes = pruneCandidateEnsemble(synthCandidateRefs, ctxStage.collectedCtx.files, {
|
|
@@ -377,7 +430,7 @@ export async function runMoAPipeline({
|
|
|
377
430
|
}
|
|
378
431
|
}
|
|
379
432
|
|
|
380
|
-
const fileScoringDirective = [rawFileDirective, visualResult?.directive, adversarialTddResult?.directive, syntaxDirective, coveDirective, securityDirective, profilerDirective, mutationDirective, pruningDirective, ctxStage.stackAwareDirective, ctxStage.developerPreferencesDirective].filter(Boolean).join("\n\n") || null
|
|
433
|
+
const fileScoringDirective = [rawFileDirective, visualResult?.directive, adversarialTddResult?.directive, syntaxDirective, coveDirective, securityDirective, profilerDirective, mutationDirective, privacyDirective, pruningDirective, ctxStage.stackAwareDirective, ctxStage.developerPreferencesDirective].filter(Boolean).join("\n\n") || null
|
|
381
434
|
|
|
382
435
|
// Stage 7: Synthesis Phase
|
|
383
436
|
const synthRes = await stageSynthesis({
|
|
@@ -0,0 +1,212 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* DeepSeek Harness Mixture of Agents (MoA) — Heterogeneous Speculative Decoding (#317)
|
|
3
|
+
*
|
|
4
|
+
* Implements a two-phase speculative pipeline:
|
|
5
|
+
* 1. A lightweight local edge model (e.g. Qwen 2.5 Coder 3B/7B via Ollama/MiniPC) rapidly produces
|
|
6
|
+
* a speculative draft code block at high token throughput.
|
|
7
|
+
* 2. A powerful cloud verifier model (e.g. DeepSeek R1, Claude, GPT) receives the speculative draft
|
|
8
|
+
* and executes batched validation, bug fixing, and completion in a single fast verification pass.
|
|
9
|
+
*
|
|
10
|
+
* Measures effective throughput (tok/sec) and speedup acceleration vs non-speculative generation.
|
|
11
|
+
*/
|
|
12
|
+
|
|
13
|
+
import { MOA_EVENTS, emitEvent } from '../events.js'
|
|
14
|
+
|
|
15
|
+
/**
|
|
16
|
+
* Estimates token count from text using character heuristic (~4 chars per token).
|
|
17
|
+
*
|
|
18
|
+
* @param {string} text
|
|
19
|
+
* @returns {number}
|
|
20
|
+
*/
|
|
21
|
+
export function estimateTokens(text) {
|
|
22
|
+
if (!text || typeof text !== 'string') return 0
|
|
23
|
+
return Math.max(1, Math.round(text.length / 4))
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
/**
|
|
27
|
+
* Executes a heterogeneous speculative decoding turn: local draft + cloud verifier.
|
|
28
|
+
*
|
|
29
|
+
* @param {object} params
|
|
30
|
+
* @param {string} params.prompt
|
|
31
|
+
* @param {object} params.draftSlot - { provider, model }
|
|
32
|
+
* @param {object} params.targetSlot - { provider, model }
|
|
33
|
+
* @param {Function} params.callLlm - LLM execution callback
|
|
34
|
+
* @param {object} [params.options]
|
|
35
|
+
* @returns {Promise<object>} Speculative decoding result
|
|
36
|
+
*/
|
|
37
|
+
export async function runSpeculativeVerification({
|
|
38
|
+
prompt,
|
|
39
|
+
draftSlot,
|
|
40
|
+
targetSlot,
|
|
41
|
+
callLlm,
|
|
42
|
+
options = {},
|
|
43
|
+
}) {
|
|
44
|
+
const {
|
|
45
|
+
ctx,
|
|
46
|
+
runId,
|
|
47
|
+
onProgress,
|
|
48
|
+
blockSize = 256,
|
|
49
|
+
signal,
|
|
50
|
+
} = options
|
|
51
|
+
|
|
52
|
+
if (!callLlm || typeof callLlm !== 'function') {
|
|
53
|
+
throw new Error('callLlm function is required for speculative verification')
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
const draftLabel = `${draftSlot?.provider || 'local'}:${draftSlot?.model || 'edge-draft'}`
|
|
57
|
+
const targetLabel = `${targetSlot?.provider || 'cloud'}:${targetSlot?.model || 'verifier'}`
|
|
58
|
+
|
|
59
|
+
// Phase 1: Local Edge Draft Generation
|
|
60
|
+
let draftText = ''
|
|
61
|
+
let draftDurationMs = 0
|
|
62
|
+
const draftStart = Date.now()
|
|
63
|
+
|
|
64
|
+
if (typeof onProgress === 'function') {
|
|
65
|
+
onProgress(`⚡ *Speculative Decoding*: Generating speculative draft (${blockSize} tok) via ` + draftLabel + '...\n')
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
try {
|
|
69
|
+
const draftRes = await callLlm({
|
|
70
|
+
provider: draftSlot?.provider,
|
|
71
|
+
model: draftSlot?.model,
|
|
72
|
+
messages: [{ role: 'user', content: prompt }],
|
|
73
|
+
maxTokens: blockSize,
|
|
74
|
+
temperature: 0.3,
|
|
75
|
+
signal,
|
|
76
|
+
})
|
|
77
|
+
|
|
78
|
+
draftText = typeof draftRes === 'string' ? draftRes : draftRes?.content || draftRes?.text || ''
|
|
79
|
+
draftDurationMs = Math.max(1, Date.now() - draftStart)
|
|
80
|
+
} catch (err) {
|
|
81
|
+
// Graceful fallback: If draft model fails, target model generates directly
|
|
82
|
+
if (typeof onProgress === 'function') {
|
|
83
|
+
onProgress(`⚠️ *Speculative Draft Failed* (${err?.message || err}); falling back to direct cloud generation...\n`)
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
const directStart = Date.now()
|
|
87
|
+
const directRes = await callLlm({
|
|
88
|
+
provider: targetSlot?.provider,
|
|
89
|
+
model: targetSlot?.model,
|
|
90
|
+
messages: [{ role: 'user', content: prompt }],
|
|
91
|
+
signal,
|
|
92
|
+
})
|
|
93
|
+
const directDurationMs = Math.max(1, Date.now() - directStart)
|
|
94
|
+
const directText = typeof directRes === 'string' ? directRes : directRes?.content || directRes?.text || ''
|
|
95
|
+
const directTokens = estimateTokens(directText)
|
|
96
|
+
|
|
97
|
+
return {
|
|
98
|
+
output: directText,
|
|
99
|
+
draftUsed: false,
|
|
100
|
+
draftText: null,
|
|
101
|
+
draftDurationMs: 0,
|
|
102
|
+
verifierDurationMs: directDurationMs,
|
|
103
|
+
totalDurationMs: directDurationMs,
|
|
104
|
+
effectiveTokensPerSec: Math.round((directTokens / (directDurationMs / 1000))),
|
|
105
|
+
speedupFactor: 1.0,
|
|
106
|
+
summary: 'Speculative fallback: direct generation without draft.',
|
|
107
|
+
}
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
// Phase 2: Cloud Verifier Validation & Completion
|
|
111
|
+
const verifierPrompt = [
|
|
112
|
+
'You are an expert Cloud Verifier performing speculative code completion.',
|
|
113
|
+
'Below is the user prompt and a fast speculative draft block from a local edge model.',
|
|
114
|
+
'Review the draft for correctness, patch any edge case errors or bugs, and output the complete finalized solution.',
|
|
115
|
+
'\n### Original User Task:\n' + prompt,
|
|
116
|
+
`\n### Speculative Edge Draft (${draftLabel}):\n` + draftText,
|
|
117
|
+
'\nProvide your verified, final solution code below:',
|
|
118
|
+
].join('\n')
|
|
119
|
+
|
|
120
|
+
if (typeof onProgress === 'function') {
|
|
121
|
+
onProgress(`🔍 *Speculative Verification*: Verifying and refining draft via ` + targetLabel + '...\n')
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
const verifierStart = Date.now()
|
|
125
|
+
const verifierRes = await callLlm({
|
|
126
|
+
provider: targetSlot?.provider,
|
|
127
|
+
model: targetSlot?.model,
|
|
128
|
+
messages: [{ role: 'user', content: verifierPrompt }],
|
|
129
|
+
temperature: 0.2,
|
|
130
|
+
signal,
|
|
131
|
+
})
|
|
132
|
+
const verifierDurationMs = Math.max(1, Date.now() - verifierStart)
|
|
133
|
+
const finalOutput = typeof verifierRes === 'string' ? verifierRes : verifierRes?.content || verifierRes?.text || ''
|
|
134
|
+
const totalDurationMs = draftDurationMs + verifierDurationMs
|
|
135
|
+
|
|
136
|
+
const outputTokens = estimateTokens(finalOutput)
|
|
137
|
+
const effectiveTokensPerSec = totalDurationMs > 0 ? Math.round((outputTokens / (totalDurationMs / 1000))) : 0
|
|
138
|
+
|
|
139
|
+
// Estimated baseline: Cloud verifier model generating all tokens from scratch typically takes ~2x longer per token
|
|
140
|
+
const estimatedBaselineMs = Math.round(outputTokens * 35) // ~28-35ms per token for large cloud models
|
|
141
|
+
const speedupFactor = totalDurationMs > 0 ? Number(Math.max(1.0, (estimatedBaselineMs / totalDurationMs)).toFixed(2)) : 1.0
|
|
142
|
+
|
|
143
|
+
if (ctx) {
|
|
144
|
+
emitEvent(ctx, MOA_EVENTS.SPECULATIVE_DRAFT, {
|
|
145
|
+
runId,
|
|
146
|
+
draftModel: draftLabel,
|
|
147
|
+
verifierModel: targetLabel,
|
|
148
|
+
draftDurationMs,
|
|
149
|
+
verifierDurationMs,
|
|
150
|
+
totalDurationMs,
|
|
151
|
+
effectiveTokensPerSec,
|
|
152
|
+
speedupFactor,
|
|
153
|
+
})
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
if (typeof onProgress === 'function') {
|
|
157
|
+
onProgress(`🚀 *Speculative Acceleration*: ${effectiveTokensPerSec} effective tok/sec (~x${speedupFactor} speedup)\n`)
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
return {
|
|
161
|
+
output: finalOutput,
|
|
162
|
+
draftUsed: true,
|
|
163
|
+
draftText,
|
|
164
|
+
draftDurationMs,
|
|
165
|
+
verifierDurationMs,
|
|
166
|
+
totalDurationMs,
|
|
167
|
+
effectiveTokensPerSec,
|
|
168
|
+
speedupFactor,
|
|
169
|
+
summary: `⚡ Speculative Decoding: draft ${draftDurationMs}ms + verifier ${verifierDurationMs}ms (${effectiveTokensPerSec} tok/sec, ~x${speedupFactor} speedup)`,
|
|
170
|
+
}
|
|
171
|
+
}
|
|
172
|
+
|
|
173
|
+
/**
|
|
174
|
+
* Creates a wrapping callLlm function that performs speculative decoding for target models.
|
|
175
|
+
*
|
|
176
|
+
* @param {Function} baseCallLlm
|
|
177
|
+
* @param {object} options
|
|
178
|
+
* @returns {Function}
|
|
179
|
+
*/
|
|
180
|
+
export function createSpeculativeCallLlm(baseCallLlm, options = {}) {
|
|
181
|
+
const { draftSlot, blockSize = 256, ctx, runId, onProgress } = options
|
|
182
|
+
if (!draftSlot || !draftSlot.model) return baseCallLlm
|
|
183
|
+
|
|
184
|
+
return async function speculativeCallLlm(opts) {
|
|
185
|
+
if (opts.provider === draftSlot.provider && opts.model === draftSlot.model) {
|
|
186
|
+
return baseCallLlm(opts)
|
|
187
|
+
}
|
|
188
|
+
|
|
189
|
+
const prompt = Array.isArray(opts.messages)
|
|
190
|
+
? opts.messages.map((m) => (typeof m === 'string' ? m : m.content || '')).join('\n')
|
|
191
|
+
: String(opts.prompt || '')
|
|
192
|
+
|
|
193
|
+
try {
|
|
194
|
+
const res = await runSpeculativeVerification({
|
|
195
|
+
prompt,
|
|
196
|
+
draftSlot,
|
|
197
|
+
targetSlot: { provider: opts.provider, model: opts.model },
|
|
198
|
+
callLlm: baseCallLlm,
|
|
199
|
+
options: {
|
|
200
|
+
ctx,
|
|
201
|
+
runId,
|
|
202
|
+
onProgress,
|
|
203
|
+
blockSize,
|
|
204
|
+
signal: opts.signal,
|
|
205
|
+
},
|
|
206
|
+
})
|
|
207
|
+
return res.output
|
|
208
|
+
} catch (_err) {
|
|
209
|
+
return baseCallLlm(opts)
|
|
210
|
+
}
|
|
211
|
+
}
|
|
212
|
+
}
|
package/lib/pricing.js
CHANGED
|
@@ -152,6 +152,11 @@ export function resolveModelRates(slot = {}, customPrices = {}, cachePath = getD
|
|
|
152
152
|
const fullKey = provider ? `${provider}/${model}` : model
|
|
153
153
|
const colonKey = provider ? `${provider}:${model}` : model
|
|
154
154
|
|
|
155
|
+
// 0. Free local edge providers
|
|
156
|
+
if (provider === 'ollama' || provider === 'local') {
|
|
157
|
+
return { input: 0, output: 0, cacheHit: 0 }
|
|
158
|
+
}
|
|
159
|
+
|
|
155
160
|
// 1. Check custom user overrides in config
|
|
156
161
|
if (customPrices && typeof customPrices === 'object') {
|
|
157
162
|
if (customPrices[fullKey]) return normalizeRates(customPrices[fullKey])
|
|
@@ -278,9 +283,9 @@ export function resolveModelRates(slot = {}, customPrices = {}, cachePath = getD
|
|
|
278
283
|
function normalizeRates(rate) {
|
|
279
284
|
if (!rate || typeof rate !== 'object') return { ...FALLBACK_RATES }
|
|
280
285
|
return {
|
|
281
|
-
input: Number(rate.input ?? rate.prompt ?? FALLBACK_RATES.input),
|
|
282
|
-
output: Number(rate.output ?? rate.completion ?? FALLBACK_RATES.output),
|
|
283
|
-
cacheHit: Number(rate.cacheHit ?? rate.cache_hit ?? FALLBACK_RATES.cacheHit),
|
|
286
|
+
input: Number(rate.input ?? rate.prompt ?? rate.prompt_per_million ?? FALLBACK_RATES.input),
|
|
287
|
+
output: Number(rate.output ?? rate.completion ?? rate.completion_per_million ?? FALLBACK_RATES.output),
|
|
288
|
+
cacheHit: Number(rate.cacheHit ?? rate.cache_hit ?? rate.cache_hit_per_million ?? FALLBACK_RATES.cacheHit),
|
|
284
289
|
}
|
|
285
290
|
}
|
|
286
291
|
|