@goodandready/dsh-moa 0.2.75 → 0.2.77
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +4 -0
- package/README.ru.md +4 -0
- package/README.zh.md +4 -0
- package/lib/client.js +102 -2
- package/lib/events.js +4 -1
- package/lib/moa-schema.js +18 -1
- package/lib/pipeline/cove-verifier.js +401 -0
- package/lib/pipeline/mutation-tester.js +323 -0
- package/lib/pipeline/runner.js +72 -2
- package/lib/pipeline/runtime-profiler.js +270 -0
- package/lib/pipeline/security-fuzzer.js +290 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -52,6 +52,10 @@ Single-model AI generation often suffers from blind spots, single-perspective bi
|
|
|
52
52
|
9. **Run History & Win-Rate Leaderboard**: Persistent logging of every run kind (synthesis, fast mode, questionnaire) with built-in REST endpoints (`/dsh-moa/history`, `/dsh-moa/leaderboard`, `/dsh-moa/runs/<id>`).
|
|
53
53
|
10. **Live Canvas 1-Click Preview (optional)**: when the `@goodandready/dsh-live-canvas` plugin is installed in the same profile, the promoted HTML is pushed to its sandbox and the MoA answer carries a one-click preview link; without it the step is skipped silently.
|
|
54
54
|
11. **Interactive Workbench (File Scoring 1–10, User Pick & Mix, Top-Model Glue & Judge Re-Check)**: Judge scores each candidate file (1–10) with pros and cons. Users can mix and match files across proposals in the WebUI. The turn's top-performing reference model harmonizes cross-file imports and contracts, followed by a final judge re-audit before promotion. Companion integration with [`dsh-time-machine`](https://github.com/goodandready/dsh-time-machine) for safe pre-promotion rollback checkpoints.
|
|
55
|
+
12. **Adversarial White-Hat Security Fuzzing Gate (#302)**: Red Team security audit scanning candidate code for SQLi, XSS, Command Injection, Path Traversal, ReDoS, and Prototype Pollution with blocking `SECURITY_VETO`.
|
|
56
|
+
13. **Chain-of-Verification (CoVe) Fact-Checking Gate for Code (#314)**: Independent pre-judge verification formulating factual API questions, validating modules and functions against standard library and project manifests to eliminate hallucinated methods.
|
|
57
|
+
14. **Runtime CPU & Memory Profiler Gate (#315)**: Sandboxed micro-benchmarking of candidate execution duration, ops/sec throughput, and heap usage with hard timeout protection to select algorithmically optimal code.
|
|
58
|
+
15. **Mutation Testing Gate for Candidate Tests (#316)**: Injects AST-level fault mutations to compute Mutation Score (killed vs survived mutants), weeding out tautological assertions and fake test suites.
|
|
55
59
|
|
|
56
60
|
---
|
|
57
61
|
|
package/README.ru.md
CHANGED
|
@@ -52,6 +52,10 @@
|
|
|
52
52
|
9. **История запусков и лидерборд моделей**: Персистентное логирование всех видов запусков (синтез, fast mode, опросник) и REST-эндпоинты (`/dsh-moa/history`, `/dsh-moa/leaderboard`, `/dsh-moa/runs/<id>`).
|
|
53
53
|
10. **Live Canvas 1-клик предпросмотр (опционально)**: если в профиле установлен `@goodandready/dsh-live-canvas`, промоученный HTML отправляется в его песочницу, а ответ MoA содержит ссылку на предпросмотр в 1 клик; без плагина шаг тихо пропускается.
|
|
54
54
|
11. **Интерактивный воркбенч (Оценка файлов 1–10, выбор в UI, склейка лучшей моделью и повторный аудит)**: Судья оценивает каждый сгенерированный файл каждого кандидата (1–10) с указанием ключевых плюсов и минусов. Пользователь комбинирует желаемый набор файлов в WebUI; лучшая модель раунда вызывается для гармонизации импортов и контрактов, после чего судья выполняет финальную валидацию перед промоушеном. Совместимо с [`dsh-time-machine`](https://github.com/goodandready/dsh-time-machine) для безопасных теневых снимков Git.
|
|
55
|
+
12. **Состязательный White-Hat фильтр безопасности (#302)**: Red Team аудит безопасности перед арбитражем судьи, выявляющий SQLi, XSS, инъекции команд, Path Traversal, ReDoS и Prototype Pollution с блокирующим вердиктом `SECURITY_VETO`.
|
|
56
|
+
13. **Фильтр верификации фактов и API (CoVe) (#314)**: Автономная цепочка верификации (Chain-of-Verification), формулирующая проверочные вопросы по вызовам методов и пакетам, исключающая галлюцинированные и несуществующие API библиотек.
|
|
57
|
+
14. **Профайлер производительности CPU и памяти (#315)**: Изолированный бенчмаркинг времени выполнения кандидатов, пропускной способности (ops/sec) и потребления heap-памяти с защитой от зависаний для выбора алгоритмически оптимального кода.
|
|
58
|
+
15. **Мутационное тестирование тестов кандидатов (#316)**: Внедрение AST-мутаций кода для расчёта Mutation Score (убитые vs выжившие мутанты), устраняющее ложноположительные и тавтологические тесты.
|
|
55
59
|
|
|
56
60
|
---
|
|
57
61
|
|
package/README.zh.md
CHANGED
|
@@ -52,6 +52,10 @@
|
|
|
52
52
|
9. **运行历史与胜率排行榜**:对每一类运行(综合、快速模式、问卷)进行持久化记录,并内置 REST 端点(`/dsh-moa/history`、`/dsh-moa/leaderboard`、`/dsh-moa/runs/<id>`)。
|
|
53
53
|
10. **Live Canvas 一键预览(可选)**:当同一 profile 中安装了 `@goodandready/dsh-live-canvas` 时,提升到项目根目录的 HTML 会被推入其沙箱,MoA 回答附带一键预览链接;未安装时该步骤静默跳过。
|
|
54
54
|
11. **交互式工作台 (1-10分文件打分、自由混搭、最优模型粘合与裁判复审)**:裁判对每个候选模型生成的每个文件进行 1–10 分打分并列出优缺点。用户可在 WebUI 中自由勾选组合所需文件;当轮最高分模型被调用以粘合协同跨文件导入与类型接口,随后由裁判进行最终复审。支持配套插件 [`dsh-time-machine`](https://github.com/goodandready/dsh-time-machine) 在文件提升前创建影子 Git 回滚快照。
|
|
55
|
+
12. **对抗性白帽安全模糊测试关卡 (#302)**:红队安全审计器在裁判前检查 SQLi、XSS、命令注入、路径穿越、ReDoS 和原型污染漏洞,发现严重漏洞时触发 `SECURITY_VETO` 否决。
|
|
56
|
+
13. **代码事实与 API 验证链关卡 (CoVe) (#314)**:独立的裁判前事实核验,生成针对性 API 问题并对照标准库及项目配置清单,杜绝调用虚构不存在的幻觉方法。
|
|
57
|
+
14. **运行期 CPU 与内存性能分析关卡 (#315)**:在隔离沙箱中微基准测试候选方案执行延迟、吞吐量(ops/sec)与堆内存占用,带有硬超时保护,指引裁判挑选算法最优代码。
|
|
58
|
+
15. **候选测试套件变异测试关卡 (#316)**:注入 AST 级别故障变异以计算变异得分(消灭 vs 存活变异体),剔除恒真废测试与虚假测试套件。
|
|
55
59
|
|
|
56
60
|
---
|
|
57
61
|
|
package/lib/client.js
CHANGED
|
@@ -228,7 +228,15 @@ window.__ModuleLoader__.load({
|
|
|
228
228
|
"aggregator.stack_personas_title": "\u{1F4E6} Stack-Aware Dynamic Persona Evolution (Project Manifests)",
|
|
229
229
|
"aggregator.stack_personas_desc": "Scans project manifests (package.json, Cargo.toml, pyproject.toml, go.mod) and assigns specialized engineering roles to candidate slots.",
|
|
230
230
|
"aggregator.preferences_title": "\u{1F9E0} Developer Coding Principles & Preferences Metaprompt",
|
|
231
|
-
"aggregator.preferences_desc": "Injects learned developer preferences (strict typing, functional immutability, error boundaries) to guide Judge synthesis."
|
|
231
|
+
"aggregator.preferences_desc": "Injects learned developer preferences (strict typing, functional immutability, error boundaries) to guide Judge synthesis.",
|
|
232
|
+
"aggregator.security_gate_title": "\u{1F6E1}\uFE0F Adversarial White-Hat Security Fuzzing Gate",
|
|
233
|
+
"aggregator.security_gate_desc": "Red-team security auditor checks candidate proposals for SQLi, XSS, ReDoS, Command Injection and Path Traversal with blocking SECURITY_VETO.",
|
|
234
|
+
"aggregator.cove_gate_title": "\u{1F50D} Chain-of-Verification (CoVe) Fact-Checking Gate",
|
|
235
|
+
"aggregator.cove_gate_desc": "Formulates API existence & method signature verification questions, disqualifying candidates that use hallucinated methods.",
|
|
236
|
+
"aggregator.profiler_gate_title": "\u26A1 Automated Runtime Memory & CPU Profiler Gate",
|
|
237
|
+
"aggregator.profiler_gate_desc": "Runs isolated micro-benchmarks measuring execution latency, throughput (ops/sec), and heap memory usage.",
|
|
238
|
+
"aggregator.mutation_gate_title": "\u{1F9EC} Runtime Mutation Testing Gate for Candidate Tests",
|
|
239
|
+
"aggregator.mutation_gate_desc": "Injects synthetic faults into candidate code to test test-suite robustness, eliminating tautological and weak tests."
|
|
232
240
|
};
|
|
233
241
|
const zh = {
|
|
234
242
|
title: "\u667A\u80FD\u4F53\u6DF7\u5408 (MoA)",
|
|
@@ -447,7 +455,15 @@ window.__ModuleLoader__.load({
|
|
|
447
455
|
"aggregator.stack_personas_title": "\u{1F4E6} \u57FA\u4E8E\u9879\u76EE\u914D\u7F6E\u6E05\u5355\u7684\u611F\u77E5\u89D2\u8272\u6F14\u5316 (Stack-Aware Personas)",
|
|
448
456
|
"aggregator.stack_personas_desc": "\u81EA\u52A8\u626B\u63CF\u9879\u76EE\u6E05\u5355 (package.json, Cargo.toml, pyproject.toml, go.mod) \u5E76\u4E3A\u5404\u5019\u9009\u69FD\u4F4D\u6307\u6D3E\u4E13\u5C5E\u4E13\u4E1A\u89D2\u8272\u3002",
|
|
449
457
|
"aggregator.preferences_title": "\u{1F9E0} \u81EA\u5B66\u4E60\u5F00\u53D1\u8005\u4EE3\u7801\u504F\u597D\u5143\u63D0\u793A\u8BCD (Developer Preferences)",
|
|
450
|
-
"aggregator.preferences_desc": "\u5C06\u5F00\u53D1\u8005\u504F\u597D\uFF08\u4E25\u683C\u7C7B\u578B\u3001\u51FD\u6570\u4E0D\u53EF\u53D8\u6027\u3001\u9632\u5FA1\u6027\u9519\u8BEF\u5904\u7406\u7B49\uFF09\u6CE8\u5165\u88C1\u5224\u7EFC\u5408\u8BC4\u5BA1\u9636\u6BB5\u3002"
|
|
458
|
+
"aggregator.preferences_desc": "\u5C06\u5F00\u53D1\u8005\u504F\u597D\uFF08\u4E25\u683C\u7C7B\u578B\u3001\u51FD\u6570\u4E0D\u53EF\u53D8\u6027\u3001\u9632\u5FA1\u6027\u9519\u8BEF\u5904\u7406\u7B49\uFF09\u6CE8\u5165\u88C1\u5224\u7EFC\u5408\u8BC4\u5BA1\u9636\u6BB5\u3002",
|
|
459
|
+
"aggregator.security_gate_title": "\u{1F6E1}\uFE0F \u5BF9\u6297\u6027\u767D\u5E3D\u5B89\u5168\u6A21\u7CCA\u6D4B\u8BD5\u5173\u5361",
|
|
460
|
+
"aggregator.security_gate_desc": "\u7EA2\u961F\u5B89\u5168\u5BA1\u8BA1\u5668\u68C0\u67E5\u5019\u9009\u65B9\u6848\u4E2D\u7684 SQLi\u3001XSS\u3001ReDoS\u3001\u547D\u4EE4\u6CE8\u5165\u548C\u8DEF\u5F84\u7A7F\u8D8A\u6F0F\u6D1E\uFF0C\u5E76\u5728\u53D1\u73B0\u4E25\u91CD\u6F0F\u6D1E\u65F6\u89E6\u53D1 SECURITY_VETO \u5426\u51B3\u3002",
|
|
461
|
+
"aggregator.cove_gate_title": "\u{1F50D} \u9A8C\u8BC1\u94FE (CoVe) \u4EE3\u7801\u4E8B\u5B9E\u4E0E API \u6838\u67E5\u5173\u5361",
|
|
462
|
+
"aggregator.cove_gate_desc": "\u751F\u6210 API \u5B58\u5728\u6027\u4E0E\u65B9\u6CD5\u7B7E\u540D\u6838\u9A8C\u95EE\u9898\uFF0C\u81EA\u52A8\u60E9\u7F5A\u6216\u53D6\u6D88\u4F7F\u7528\u865A\u6784\u5E7B\u89C9\u65B9\u6CD5\u7684\u5019\u9009\u65B9\u6848\u3002",
|
|
463
|
+
"aggregator.profiler_gate_title": "\u26A1 \u81EA\u52A8\u8FD0\u884C\u65F6 CPU \u4E0E\u5185\u5B58\u6027\u80FD\u5206\u6790\u5173\u5361",
|
|
464
|
+
"aggregator.profiler_gate_desc": "\u5728\u9694\u79BB\u6C99\u7BB1\u4E2D\u8FD0\u884C\u5FAE\u57FA\u51C6\u6D4B\u8BD5\uFF0C\u7CBE\u786E\u5EA6\u91CF\u6267\u884C\u5EF6\u8FDF\u3001\u541E\u5410\u91CF (ops/sec) \u548C\u5806\u5185\u5B58\u5206\u914D\u3002",
|
|
465
|
+
"aggregator.mutation_gate_title": "\u{1F9EC} \u5019\u9009\u6D4B\u8BD5\u5957\u4EF6\u8FD0\u884C\u65F6\u53D8\u5F02\u6D4B\u8BD5\u5173\u5361",
|
|
466
|
+
"aggregator.mutation_gate_desc": "\u5411\u4EE3\u7801\u6CE8\u5165\u5408\u6210\u6545\u969C\u5E76\u9A8C\u8BC1\u6D4B\u8BD5\u7684\u6355\u83B7\u80FD\u529B\uFF0C\u5254\u9664\u540C\u4E49\u53CD\u590D\u4E0E\u65E0\u6548\u7684\u865A\u5047\u6D4B\u8BD5\u3002"
|
|
451
467
|
};
|
|
452
468
|
function makeT(dict, fallback) {
|
|
453
469
|
return function t(key, vars) {
|
|
@@ -2610,6 +2626,90 @@ window.__ModuleLoader__.load({
|
|
|
2610
2626
|
t("aggregator.preferences_desc") || "Injects learned developer preferences (strict typing, functional immutability, error boundaries) to guide Judge synthesis."
|
|
2611
2627
|
)
|
|
2612
2628
|
),
|
|
2629
|
+
/* Adversarial White-Hat Security Fuzzing Gate (#302) */
|
|
2630
|
+
React.createElement(
|
|
2631
|
+
"div",
|
|
2632
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6, padding: "8px 12px", background: "var(--dsw-alias-bg-layer-2)", borderRadius: 8, border: "1px solid var(--dsw-alias-border-l2)", marginTop: 8 } },
|
|
2633
|
+
React.createElement(
|
|
2634
|
+
"label",
|
|
2635
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, cursor: "pointer", fontSize: 13, fontWeight: 500 } },
|
|
2636
|
+
React.createElement("input", {
|
|
2637
|
+
type: "checkbox",
|
|
2638
|
+
checked: Boolean(currentPreset.security_gate_enabled),
|
|
2639
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, security_gate_enabled: e.target.checked })),
|
|
2640
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2641
|
+
}),
|
|
2642
|
+
t("aggregator.security_gate_title") || "\u{1F6E1}\uFE0F Adversarial White-Hat Security Fuzzing Gate"
|
|
2643
|
+
),
|
|
2644
|
+
React.createElement(
|
|
2645
|
+
"div",
|
|
2646
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2647
|
+
t("aggregator.security_gate_desc") || "Red-team security auditor checks candidate proposals for SQLi, XSS, ReDoS, Command Injection and Path Traversal with blocking SECURITY_VETO."
|
|
2648
|
+
)
|
|
2649
|
+
),
|
|
2650
|
+
/* Chain-of-Verification (CoVe) Fact-Checking Gate (#314) */
|
|
2651
|
+
React.createElement(
|
|
2652
|
+
"div",
|
|
2653
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6, padding: "8px 12px", background: "var(--dsw-alias-bg-layer-2)", borderRadius: 8, border: "1px solid var(--dsw-alias-border-l2)", marginTop: 8 } },
|
|
2654
|
+
React.createElement(
|
|
2655
|
+
"label",
|
|
2656
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, cursor: "pointer", fontSize: 13, fontWeight: 500 } },
|
|
2657
|
+
React.createElement("input", {
|
|
2658
|
+
type: "checkbox",
|
|
2659
|
+
checked: Boolean(currentPreset.cove_verification_enabled),
|
|
2660
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, cove_verification_enabled: e.target.checked })),
|
|
2661
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2662
|
+
}),
|
|
2663
|
+
t("aggregator.cove_gate_title") || "\u{1F50D} Chain-of-Verification (CoVe) Fact-Checking Gate"
|
|
2664
|
+
),
|
|
2665
|
+
React.createElement(
|
|
2666
|
+
"div",
|
|
2667
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2668
|
+
t("aggregator.cove_gate_desc") || "Formulates API existence & method signature verification questions, disqualifying candidates that use hallucinated methods."
|
|
2669
|
+
)
|
|
2670
|
+
),
|
|
2671
|
+
/* Automated Runtime Memory & CPU Profiler Gate (#315) */
|
|
2672
|
+
React.createElement(
|
|
2673
|
+
"div",
|
|
2674
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6, padding: "8px 12px", background: "var(--dsw-alias-bg-layer-2)", borderRadius: 8, border: "1px solid var(--dsw-alias-border-l2)", marginTop: 8 } },
|
|
2675
|
+
React.createElement(
|
|
2676
|
+
"label",
|
|
2677
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, cursor: "pointer", fontSize: 13, fontWeight: 500 } },
|
|
2678
|
+
React.createElement("input", {
|
|
2679
|
+
type: "checkbox",
|
|
2680
|
+
checked: Boolean(currentPreset.profiler_gate_enabled),
|
|
2681
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, profiler_gate_enabled: e.target.checked })),
|
|
2682
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2683
|
+
}),
|
|
2684
|
+
t("aggregator.profiler_gate_title") || "\u26A1 Automated Runtime Memory & CPU Profiler Gate"
|
|
2685
|
+
),
|
|
2686
|
+
React.createElement(
|
|
2687
|
+
"div",
|
|
2688
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2689
|
+
t("aggregator.profiler_gate_desc") || "Runs isolated micro-benchmarks measuring execution latency, throughput (ops/sec), and heap memory usage."
|
|
2690
|
+
)
|
|
2691
|
+
),
|
|
2692
|
+
/* Runtime Mutation Testing Gate (#316) */
|
|
2693
|
+
React.createElement(
|
|
2694
|
+
"div",
|
|
2695
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6, padding: "8px 12px", background: "var(--dsw-alias-bg-layer-2)", borderRadius: 8, border: "1px solid var(--dsw-alias-border-l2)", marginTop: 8 } },
|
|
2696
|
+
React.createElement(
|
|
2697
|
+
"label",
|
|
2698
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, cursor: "pointer", fontSize: 13, fontWeight: 500 } },
|
|
2699
|
+
React.createElement("input", {
|
|
2700
|
+
type: "checkbox",
|
|
2701
|
+
checked: Boolean(currentPreset.mutation_testing_enabled),
|
|
2702
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, mutation_testing_enabled: e.target.checked })),
|
|
2703
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2704
|
+
}),
|
|
2705
|
+
t("aggregator.mutation_gate_title") || "\u{1F9EC} Runtime Mutation Testing Gate for Candidate Tests"
|
|
2706
|
+
),
|
|
2707
|
+
React.createElement(
|
|
2708
|
+
"div",
|
|
2709
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2710
|
+
t("aggregator.mutation_gate_desc") || "Injects synthetic faults into candidate code to test test-suite robustness, eliminating tautological and weak tests."
|
|
2711
|
+
)
|
|
2712
|
+
),
|
|
2613
2713
|
/* Fallback Judges Chain */
|
|
2614
2714
|
React.createElement(
|
|
2615
2715
|
"div",
|
package/lib/events.js
CHANGED
|
@@ -11,6 +11,10 @@ export const MOA_EVENTS = {
|
|
|
11
11
|
SELF_CONSISTENCY_EXPAND: 'moa/self-consistency-expand',
|
|
12
12
|
STACK_DETECTED: 'moa/stack-detected',
|
|
13
13
|
PREFERENCES_APPLIED: 'moa/preferences-applied',
|
|
14
|
+
SECURITY_AUDIT: 'moa/security-audit',
|
|
15
|
+
COVE_VERIFIED: 'moa/cove-verified',
|
|
16
|
+
PROFILER_COMPLETE: 'moa/profiler-complete',
|
|
17
|
+
MUTATION_COMPLETE: 'moa/mutation-complete',
|
|
14
18
|
CRITIQUE_COMPLETE: 'moa/critique-complete',
|
|
15
19
|
JUDGE_VERDICT: 'moa/judge-verdict',
|
|
16
20
|
CONTEXT_PRUNING: 'moa/context-pruning',
|
|
@@ -36,4 +40,3 @@ export function emitEvent(ctx, eventName, payload) {
|
|
|
36
40
|
/* best-effort event dispatch: listener exceptions must never abort pipeline */
|
|
37
41
|
}
|
|
38
42
|
}
|
|
39
|
-
|
package/lib/moa-schema.js
CHANGED
|
@@ -91,6 +91,15 @@ export const PresetSchema = z.object({
|
|
|
91
91
|
escalation_judge: ModelSlotSchema.default({ provider: '', model: '' }),
|
|
92
92
|
stack_aware_personas_enabled: z.boolean().default(false),
|
|
93
93
|
developer_preferences_enabled: z.boolean().default(false),
|
|
94
|
+
security_gate_enabled: z.boolean().default(false),
|
|
95
|
+
security_gate_strict_veto: z.boolean().default(true),
|
|
96
|
+
cove_verification_enabled: z.boolean().default(false),
|
|
97
|
+
cove_max_questions: z.number().default(5),
|
|
98
|
+
profiler_gate_enabled: z.boolean().default(false),
|
|
99
|
+
profiler_iterations: z.number().default(100),
|
|
100
|
+
profiler_timeout_ms: z.number().default(2000),
|
|
101
|
+
mutation_testing_enabled: z.boolean().default(false),
|
|
102
|
+
min_mutation_score: z.number().default(0.70),
|
|
94
103
|
})
|
|
95
104
|
|
|
96
105
|
export const Config = z.object({
|
|
@@ -128,6 +137,15 @@ export const Config = z.object({
|
|
|
128
137
|
escalation_judge: ModelSlotSchema.default({ provider: '', model: '' }).volatile(),
|
|
129
138
|
stack_aware_personas_enabled: z.boolean().default(false).volatile(),
|
|
130
139
|
developer_preferences_enabled: z.boolean().default(false).volatile(),
|
|
140
|
+
security_gate_enabled: z.boolean().default(false).volatile(),
|
|
141
|
+
security_gate_strict_veto: z.boolean().default(true).volatile(),
|
|
142
|
+
cove_verification_enabled: z.boolean().default(false).volatile(),
|
|
143
|
+
cove_max_questions: z.number().default(5).volatile(),
|
|
144
|
+
profiler_gate_enabled: z.boolean().default(false).volatile(),
|
|
145
|
+
profiler_iterations: z.number().default(100).volatile(),
|
|
146
|
+
profiler_timeout_ms: z.number().default(2000).volatile(),
|
|
147
|
+
mutation_testing_enabled: z.boolean().default(false).volatile(),
|
|
148
|
+
min_mutation_score: z.number().default(0.70).volatile(),
|
|
131
149
|
smart_routing_model: ModelSlotSchema.default({ provider: '', model: '' }).volatile(),
|
|
132
150
|
prices: z.dict(PriceRow).default({}).volatile(),
|
|
133
151
|
presets: z.array(PresetSchema).default([
|
|
@@ -197,4 +215,3 @@ export function plainConfig(value) {
|
|
|
197
215
|
}
|
|
198
216
|
return value
|
|
199
217
|
}
|
|
200
|
-
|
|
@@ -0,0 +1,401 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* DeepSeek Harness Mixture of Agents (MoA) — Chain-of-Verification (CoVe) Fact-Checking Gate (#314)
|
|
3
|
+
*
|
|
4
|
+
* Pre-Judge verification gate that analyzes candidate proposals, formulates 3-5 factual API & syntax
|
|
5
|
+
* verification questions, audits them against standard libraries & project context, and flags
|
|
6
|
+
* hallucinated methods or nonexistent modules before arbitration.
|
|
7
|
+
*/
|
|
8
|
+
|
|
9
|
+
import { MOA_EVENTS, emitEvent } from '../events.js'
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* Standard Node.js built-in modules.
|
|
13
|
+
*/
|
|
14
|
+
export const NODE_STDLIB_MODULES = new Set([
|
|
15
|
+
'assert', 'async_hooks', 'buffer', 'child_process', 'cluster', 'console',
|
|
16
|
+
'constants', 'crypto', 'dgram', 'diagnostics_channel', 'dns', 'domain',
|
|
17
|
+
'events', 'fs', 'fs/promises', 'http', 'http2', 'https', 'inspector',
|
|
18
|
+
'module', 'net', 'os', 'path', 'path/posix', 'path/win32', 'perf_hooks',
|
|
19
|
+
'process', 'punycode', 'querystring', 'readline', 'repl', 'stream',
|
|
20
|
+
'stream/promises', 'stream/web', 'string_decoder', 'test', 'timers',
|
|
21
|
+
'timers/promises', 'tls', 'trace_events', 'tty', 'url', 'util',
|
|
22
|
+
'v8', 'vm', 'wasi', 'worker_threads', 'zlib',
|
|
23
|
+
])
|
|
24
|
+
|
|
25
|
+
/**
|
|
26
|
+
* Common hallucinated APIs and their real replacements.
|
|
27
|
+
*/
|
|
28
|
+
export const KNOWN_HALLUCINATIONS = [
|
|
29
|
+
{
|
|
30
|
+
regex: /fs\.(?:readAllText|readTextFile|readText|readAsString)\s*\(/i,
|
|
31
|
+
hallucination: 'fs.readAllText / fs.readTextFile',
|
|
32
|
+
correction: 'fs.readFileSync(path, "utf8") or fs.promises.readFile(path, "utf8")',
|
|
33
|
+
reason: 'Method does not exist in Node.js fs module (often hallucinated from C# .NET or Deno).',
|
|
34
|
+
},
|
|
35
|
+
{
|
|
36
|
+
regex: /fs\.(?:writeAllText|writeTextFile|writeText|writeAsString)\s*\(/i,
|
|
37
|
+
hallucination: 'fs.writeAllText / fs.writeTextFile',
|
|
38
|
+
correction: 'fs.writeFileSync(path, data, "utf8") or fs.promises.writeFile(path, data, "utf8")',
|
|
39
|
+
reason: 'Method does not exist in Node.js fs module.',
|
|
40
|
+
},
|
|
41
|
+
{
|
|
42
|
+
regex: /fs\.readFileAsync\s*\(/i,
|
|
43
|
+
hallucination: 'fs.readFileAsync',
|
|
44
|
+
correction: 'fs.promises.readFile(...) or util.promisify(fs.readFile)',
|
|
45
|
+
reason: 'Legacy Bluebird promise naming is not a native Node.js fs method.',
|
|
46
|
+
},
|
|
47
|
+
{
|
|
48
|
+
regex: /path\.(?:normalizeUrl|combine|joinPath)\s*\(/i,
|
|
49
|
+
hallucination: 'path.combine / path.normalizeUrl',
|
|
50
|
+
correction: 'path.join(...) or new URL(...)',
|
|
51
|
+
reason: 'Method does not exist in Node.js path module.',
|
|
52
|
+
},
|
|
53
|
+
{
|
|
54
|
+
regex: /crypto\.(?:generateRandomUuidString|randomString|generateUuid)\s*\(/i,
|
|
55
|
+
hallucination: 'crypto.generateRandomUuidString',
|
|
56
|
+
correction: 'crypto.randomUUID() or crypto.randomBytes(...)',
|
|
57
|
+
reason: 'Nonexistent crypto helper hallucinated by language models.',
|
|
58
|
+
},
|
|
59
|
+
{
|
|
60
|
+
regex: /JSON\.(?:toObject|fromObject|encode|decode)\s*\(/i,
|
|
61
|
+
hallucination: 'JSON.toObject / JSON.fromObject / JSON.encode',
|
|
62
|
+
correction: 'JSON.parse(...) or JSON.stringify(...)',
|
|
63
|
+
reason: 'Invalid JSON global methods hallucinated from other ecosystems.',
|
|
64
|
+
},
|
|
65
|
+
{
|
|
66
|
+
regex: /Promise\.(?:wait|sleep|delay)\s*\(/i,
|
|
67
|
+
hallucination: 'Promise.sleep / Promise.wait / Promise.delay',
|
|
68
|
+
correction: 'new Promise(resolve => setTimeout(resolve, ms)) or timers/promises setTimeout',
|
|
69
|
+
reason: 'Native JavaScript Promise does not feature sleep/delay directly on Promise constructor.',
|
|
70
|
+
},
|
|
71
|
+
]
|
|
72
|
+
|
|
73
|
+
/**
|
|
74
|
+
* Extracts candidate imports, requires, and method calls to formulate verification questions.
|
|
75
|
+
*
|
|
76
|
+
* @param {string} code
|
|
77
|
+
* @param {object} [options]
|
|
78
|
+
* @returns {Array<object>} Up to maxQuestions questions
|
|
79
|
+
*/
|
|
80
|
+
export function generateCoveQuestions(code, options = {}) {
|
|
81
|
+
const maxQuestions = Math.min(10, Math.max(1, options.maxQuestions || 5))
|
|
82
|
+
const questions = []
|
|
83
|
+
if (!code || typeof code !== 'string') return questions
|
|
84
|
+
|
|
85
|
+
// 1. Check imports/requires
|
|
86
|
+
const importRegex = /(?:import\s+(?:\{[^}]+\}|\*\s+as\s+\w+|\w+)\s+from\s+['"]([^'"]+)['"]|require\s*\(\s*['"]([^'"]+)['"]\s*\))/g
|
|
87
|
+
let match
|
|
88
|
+
const seenModules = new Set()
|
|
89
|
+
while ((match = importRegex.exec(code)) !== null) {
|
|
90
|
+
const mod = match[1] || match[2]
|
|
91
|
+
if (mod && !seenModules.has(mod) && !mod.startsWith('.') && !mod.startsWith('/')) {
|
|
92
|
+
seenModules.add(mod)
|
|
93
|
+
const cleanMod = mod.replace(/^node:/, '')
|
|
94
|
+
questions.push({
|
|
95
|
+
id: questions.length + 1,
|
|
96
|
+
type: 'module_resolution',
|
|
97
|
+
subject: mod,
|
|
98
|
+
cleanSubject: cleanMod,
|
|
99
|
+
question: `Is module '${mod}' available in standard library or project dependencies?`,
|
|
100
|
+
})
|
|
101
|
+
if (questions.length >= maxQuestions) return questions
|
|
102
|
+
}
|
|
103
|
+
}
|
|
104
|
+
|
|
105
|
+
// 2. Check for known hallucination method calls
|
|
106
|
+
for (const h of KNOWN_HALLUCINATIONS) {
|
|
107
|
+
if (h.regex.test(code)) {
|
|
108
|
+
questions.push({
|
|
109
|
+
id: questions.length + 1,
|
|
110
|
+
type: 'api_existence',
|
|
111
|
+
subject: h.hallucination,
|
|
112
|
+
hallucinationDef: h,
|
|
113
|
+
question: `Does API '${h.hallucination}' exist and is it valid in the target runtime?`,
|
|
114
|
+
})
|
|
115
|
+
if (questions.length >= maxQuestions) return questions
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
// 3. Check general API calls (fs, path, crypto, child_process)
|
|
120
|
+
const apiCallRegex = /\b(fs|path|crypto|os|child_process|JSON|Promise)\.([a-zA-Z0-9_]+)\s*\(/g
|
|
121
|
+
const seenApis = new Set()
|
|
122
|
+
while ((match = apiCallRegex.exec(code)) !== null) {
|
|
123
|
+
const obj = match[1]
|
|
124
|
+
const method = match[2]
|
|
125
|
+
const full = `${obj}.${method}`
|
|
126
|
+
if (!seenApis.has(full)) {
|
|
127
|
+
seenApis.add(full)
|
|
128
|
+
questions.push({
|
|
129
|
+
id: questions.length + 1,
|
|
130
|
+
type: 'method_signature',
|
|
131
|
+
subject: full,
|
|
132
|
+
object: obj,
|
|
133
|
+
method,
|
|
134
|
+
question: `Does method '${method}' natively exist on standard global '${obj}'?`,
|
|
135
|
+
})
|
|
136
|
+
if (questions.length >= maxQuestions) return questions
|
|
137
|
+
}
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
// Fallback question if code has no special calls
|
|
141
|
+
if (questions.length === 0) {
|
|
142
|
+
questions.push({
|
|
143
|
+
id: 1,
|
|
144
|
+
type: 'syntax_coherence',
|
|
145
|
+
subject: 'code_structure',
|
|
146
|
+
question: 'Are all functions and return types syntactically aligned and verifiable?',
|
|
147
|
+
})
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
return questions.slice(0, maxQuestions)
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
/**
|
|
154
|
+
* Verifies a single CoVe factual question against stdlib and project dependencies.
|
|
155
|
+
*
|
|
156
|
+
* @param {object} q - Question object from generateCoveQuestions
|
|
157
|
+
* @param {object} [projectContext]
|
|
158
|
+
* @returns {object} Verification result
|
|
159
|
+
*/
|
|
160
|
+
export function verifyCoveQuestion(q, projectContext = {}) {
|
|
161
|
+
const deps = {
|
|
162
|
+
...(projectContext.dependencies || {}),
|
|
163
|
+
...(projectContext.devDependencies || {}),
|
|
164
|
+
...(projectContext.stackInfo?.manifestDependencies || {}),
|
|
165
|
+
}
|
|
166
|
+
|
|
167
|
+
if (q.type === 'module_resolution') {
|
|
168
|
+
const isNode = NODE_STDLIB_MODULES.has(q.cleanSubject)
|
|
169
|
+
const isDep = Boolean(deps[q.cleanSubject] || deps[q.subject])
|
|
170
|
+
if (isNode || isDep) {
|
|
171
|
+
return {
|
|
172
|
+
verified: true,
|
|
173
|
+
confidence: 1.0,
|
|
174
|
+
rationale: isNode
|
|
175
|
+
? `Module '${q.subject}' is a verified Node.js standard library module.`
|
|
176
|
+
: `Module '${q.subject}' is defined in project manifest dependencies.`,
|
|
177
|
+
hallucination: null,
|
|
178
|
+
}
|
|
179
|
+
}
|
|
180
|
+
return {
|
|
181
|
+
verified: false,
|
|
182
|
+
confidence: 0.85,
|
|
183
|
+
rationale: `Module '${q.subject}' is not found in Node.js stdlib or declared project dependencies.`,
|
|
184
|
+
hallucination: {
|
|
185
|
+
api: q.subject,
|
|
186
|
+
reason: `Module '${q.subject}' was referenced without declaration in package.json or stdlib.`,
|
|
187
|
+
},
|
|
188
|
+
}
|
|
189
|
+
}
|
|
190
|
+
|
|
191
|
+
if (q.type === 'api_existence') {
|
|
192
|
+
// Matched a known hallucination pattern
|
|
193
|
+
const h = q.hallucinationDef
|
|
194
|
+
return {
|
|
195
|
+
verified: false,
|
|
196
|
+
confidence: 0.95,
|
|
197
|
+
rationale: `Invalid API detected: ${h.reason} Suggested fix: ${h.correction}`,
|
|
198
|
+
hallucination: {
|
|
199
|
+
api: h.hallucination,
|
|
200
|
+
reason: h.reason,
|
|
201
|
+
correction: h.correction,
|
|
202
|
+
},
|
|
203
|
+
}
|
|
204
|
+
}
|
|
205
|
+
|
|
206
|
+
if (q.type === 'method_signature') {
|
|
207
|
+
// Verify common valid methods on known objects
|
|
208
|
+
const validFs = new Set(['readFile', 'readFileSync', 'writeFile', 'writeFileSync', 'existsSync', 'statSync', 'stat', 'mkdirSync', 'mkdir', 'unlinkSync', 'unlink', 'readdirSync', 'readdir', 'createReadStream', 'createWriteStream', 'promises'])
|
|
209
|
+
const validPath = new Set(['join', 'resolve', 'basename', 'dirname', 'extname', 'relative', 'normalize', 'isAbsolute', 'parse', 'format', 'sep', 'posix', 'win32'])
|
|
210
|
+
const validCrypto = new Set(['randomUUID', 'randomBytes', 'createHash', 'createHmac', 'createCipheriv', 'createDecipheriv', 'timingSafeEqual', 'scryptSync', 'pbkdf2Sync'])
|
|
211
|
+
const validJson = new Set(['parse', 'stringify'])
|
|
212
|
+
const validPromise = new Set(['all', 'allSettled', 'race', 'any', 'resolve', 'reject'])
|
|
213
|
+
|
|
214
|
+
let isValid = true
|
|
215
|
+
if (q.object === 'fs') isValid = validFs.has(q.method)
|
|
216
|
+
else if (q.object === 'path') isValid = validPath.has(q.method)
|
|
217
|
+
else if (q.object === 'crypto') isValid = validCrypto.has(q.method)
|
|
218
|
+
else if (q.object === 'JSON') isValid = validJson.has(q.method)
|
|
219
|
+
else if (q.object === 'Promise') isValid = validPromise.has(q.method)
|
|
220
|
+
|
|
221
|
+
if (isValid) {
|
|
222
|
+
return {
|
|
223
|
+
verified: true,
|
|
224
|
+
confidence: 0.95,
|
|
225
|
+
rationale: `'${q.subject}' is a verified standard method.`,
|
|
226
|
+
hallucination: null,
|
|
227
|
+
}
|
|
228
|
+
}
|
|
229
|
+
return {
|
|
230
|
+
verified: false,
|
|
231
|
+
confidence: 0.9,
|
|
232
|
+
rationale: `'${q.method}' does not exist on global '${q.object}'.`,
|
|
233
|
+
hallucination: {
|
|
234
|
+
api: q.subject,
|
|
235
|
+
reason: `Method '${q.method}' does not exist on '${q.object}'.`,
|
|
236
|
+
},
|
|
237
|
+
}
|
|
238
|
+
}
|
|
239
|
+
|
|
240
|
+
return {
|
|
241
|
+
verified: true,
|
|
242
|
+
confidence: 0.9,
|
|
243
|
+
rationale: 'Verified general syntax structure.',
|
|
244
|
+
hallucination: null,
|
|
245
|
+
}
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
/**
|
|
249
|
+
* Audits a single candidate solution using Chain-of-Verification (CoVe).
|
|
250
|
+
*
|
|
251
|
+
* @param {string} code
|
|
252
|
+
* @param {object} [projectContext]
|
|
253
|
+
* @param {object} [options]
|
|
254
|
+
* @returns {object}
|
|
255
|
+
*/
|
|
256
|
+
export function auditCandidateCove(code, projectContext = {}, options = {}) {
|
|
257
|
+
const questions = generateCoveQuestions(code, options)
|
|
258
|
+
const verifications = []
|
|
259
|
+
const hallucinations = []
|
|
260
|
+
|
|
261
|
+
for (const q of questions) {
|
|
262
|
+
const v = verifyCoveQuestion(q, projectContext)
|
|
263
|
+
verifications.push({
|
|
264
|
+
...q,
|
|
265
|
+
verified: v.verified,
|
|
266
|
+
confidence: v.confidence,
|
|
267
|
+
rationale: v.rationale,
|
|
268
|
+
})
|
|
269
|
+
if (!v.verified && v.hallucination) {
|
|
270
|
+
hallucinations.push(v.hallucination)
|
|
271
|
+
}
|
|
272
|
+
}
|
|
273
|
+
|
|
274
|
+
const penalty = hallucinations.length * 35
|
|
275
|
+
const coveScore = Math.max(0, 100 - penalty)
|
|
276
|
+
const passed = hallucinations.length === 0
|
|
277
|
+
|
|
278
|
+
let summary = ''
|
|
279
|
+
if (passed) {
|
|
280
|
+
summary = `🔍 CoVe Fact-Checking: PASSED (Score 100/100). Verified ${questions.length} factual API/module question(s).`
|
|
281
|
+
} else {
|
|
282
|
+
summary = `⚠️ CoVe Hallucination Detected (Score ${coveScore}/100): ${hallucinations.length} non-existent or unverified API(s) found.`
|
|
283
|
+
}
|
|
284
|
+
|
|
285
|
+
return {
|
|
286
|
+
passed,
|
|
287
|
+
coveScore,
|
|
288
|
+
questions: verifications,
|
|
289
|
+
hallucinations,
|
|
290
|
+
summary,
|
|
291
|
+
}
|
|
292
|
+
}
|
|
293
|
+
|
|
294
|
+
/**
|
|
295
|
+
* Runs the Chain-of-Verification Gate across all candidates.
|
|
296
|
+
*
|
|
297
|
+
* @param {Array<object>} candidates
|
|
298
|
+
* @param {object} [projectContext]
|
|
299
|
+
* @param {object} [options]
|
|
300
|
+
* @returns {object}
|
|
301
|
+
*/
|
|
302
|
+
export function runCoveVerificationGate(candidates, projectContext = {}, options = {}) {
|
|
303
|
+
const { ctx, runId, onProgress } = options
|
|
304
|
+
const results = []
|
|
305
|
+
let totalHallucinations = 0
|
|
306
|
+
|
|
307
|
+
if (!Array.isArray(candidates) || candidates.length === 0) {
|
|
308
|
+
return {
|
|
309
|
+
results: [],
|
|
310
|
+
passedCount: 0,
|
|
311
|
+
hallucinationCount: 0,
|
|
312
|
+
coveDirective: null,
|
|
313
|
+
}
|
|
314
|
+
}
|
|
315
|
+
|
|
316
|
+
for (let i = 0; i < candidates.length; i++) {
|
|
317
|
+
const cand = candidates[i]
|
|
318
|
+
const text = typeof cand === 'string' ? cand : cand.output || cand.text || cand.code || ''
|
|
319
|
+
const report = auditCandidateCove(text, projectContext, options)
|
|
320
|
+
|
|
321
|
+
if (typeof cand === 'object' && cand !== null) {
|
|
322
|
+
cand.coveReport = report
|
|
323
|
+
cand.coveScore = report.coveScore
|
|
324
|
+
cand.coveHallucinations = report.hallucinations
|
|
325
|
+
}
|
|
326
|
+
|
|
327
|
+
totalHallucinations += report.hallucinations.length
|
|
328
|
+
results.push({
|
|
329
|
+
candidateIndex: i,
|
|
330
|
+
model: cand?.model || `Candidate ${i + 1}`,
|
|
331
|
+
report,
|
|
332
|
+
})
|
|
333
|
+
|
|
334
|
+
if (ctx) {
|
|
335
|
+
emitEvent(ctx, MOA_EVENTS.COVE_VERIFIED, {
|
|
336
|
+
runId,
|
|
337
|
+
candidateIndex: i,
|
|
338
|
+
coveScore: report.coveScore,
|
|
339
|
+
passed: report.passed,
|
|
340
|
+
questionsCount: report.questions.length,
|
|
341
|
+
hallucinationsCount: report.hallucinations.length,
|
|
342
|
+
})
|
|
343
|
+
}
|
|
344
|
+
}
|
|
345
|
+
|
|
346
|
+
const passedCount = results.filter((r) => r.report.passed).length
|
|
347
|
+
|
|
348
|
+
if (typeof onProgress === 'function') {
|
|
349
|
+
if (totalHallucinations === 0) {
|
|
350
|
+
onProgress(`🔍 *CoVe Fact-Check Passed*: All APIs and modules verified against project context.\n`)
|
|
351
|
+
} else {
|
|
352
|
+
onProgress(`⚠️ *CoVe Fact-Check Alert*: Found ${totalHallucinations} unverified or hallucinated API(s) across candidate solutions.\n`)
|
|
353
|
+
}
|
|
354
|
+
}
|
|
355
|
+
|
|
356
|
+
const coveDirective = formatCoveDirective(results)
|
|
357
|
+
|
|
358
|
+
return {
|
|
359
|
+
results,
|
|
360
|
+
passedCount,
|
|
361
|
+
hallucinationCount: totalHallucinations,
|
|
362
|
+
coveDirective,
|
|
363
|
+
}
|
|
364
|
+
}
|
|
365
|
+
|
|
366
|
+
/**
|
|
367
|
+
* Formats CoVe fact-checking verification findings into a prompt directive for Judge/Synthesis.
|
|
368
|
+
*
|
|
369
|
+
* @param {Array<object>} results
|
|
370
|
+
* @returns {string|null}
|
|
371
|
+
*/
|
|
372
|
+
export function formatCoveDirective(results) {
|
|
373
|
+
if (!Array.isArray(results) || results.length === 0) return null
|
|
374
|
+
|
|
375
|
+
const hasAnyHallucinations = results.some((r) => r.report && r.report.hallucinations.length > 0)
|
|
376
|
+
if (!hasAnyHallucinations) {
|
|
377
|
+
return '### 🔍 CHAIN-OF-VERIFICATION (CoVe): ALL CANDIDATE APIS VERIFIED FACTUAL\nNo hallucinated APIs, phantom modules, or broken native signatures detected.'
|
|
378
|
+
}
|
|
379
|
+
|
|
380
|
+
const lines = [
|
|
381
|
+
'### 🔍 CHAIN-OF-VERIFICATION (CoVe) FACT-CHECK DIRECTIVE',
|
|
382
|
+
'Independent factual audit of API calls and module imports has been completed:',
|
|
383
|
+
]
|
|
384
|
+
|
|
385
|
+
for (const r of results) {
|
|
386
|
+
const label = `Candidate ${r.candidateIndex + 1}`
|
|
387
|
+
const rep = r.report || {}
|
|
388
|
+
if (!rep.passed && rep.hallucinations.length > 0) {
|
|
389
|
+
lines.push(`- **${label}**: ❌ **FACT-CHECK FAILURE** (Score ${rep.coveScore}/100) — Hallucinated APIs detected:`)
|
|
390
|
+
for (const h of rep.hallucinations) {
|
|
391
|
+
lines.push(` * \`${h.api}\`: ${h.reason}${h.correction ? ` (Correction: ${h.correction})` : ''}`)
|
|
392
|
+
}
|
|
393
|
+
} else {
|
|
394
|
+
lines.push(`- **${label}**: ✅ VERIFIED (Score 100/100) — All ${rep.questions?.length || 0} API checks confirmed valid.`)
|
|
395
|
+
}
|
|
396
|
+
}
|
|
397
|
+
|
|
398
|
+
lines.push('\n**Judge Fact-Checking Rule**: Do not award winning status to solutions that rely on hallucinated methods or nonexistent packages. Prefer solutions with 100/100 verified API accuracy.')
|
|
399
|
+
|
|
400
|
+
return lines.join('\n')
|
|
401
|
+
}
|