@goodandready/dsh-moa 0.2.75 → 0.2.76
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -0
- package/README.ru.md +2 -0
- package/README.zh.md +2 -0
- package/lib/client.js +52 -2
- package/lib/events.js +2 -1
- package/lib/moa-schema.js +8 -1
- package/lib/pipeline/cove-verifier.js +401 -0
- package/lib/pipeline/runner.js +37 -2
- package/lib/pipeline/security-fuzzer.js +290 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -52,6 +52,8 @@ Single-model AI generation often suffers from blind spots, single-perspective bi
|
|
|
52
52
|
9. **Run History & Win-Rate Leaderboard**: Persistent logging of every run kind (synthesis, fast mode, questionnaire) with built-in REST endpoints (`/dsh-moa/history`, `/dsh-moa/leaderboard`, `/dsh-moa/runs/<id>`).
|
|
53
53
|
10. **Live Canvas 1-Click Preview (optional)**: when the `@goodandready/dsh-live-canvas` plugin is installed in the same profile, the promoted HTML is pushed to its sandbox and the MoA answer carries a one-click preview link; without it the step is skipped silently.
|
|
54
54
|
11. **Interactive Workbench (File Scoring 1–10, User Pick & Mix, Top-Model Glue & Judge Re-Check)**: Judge scores each candidate file (1–10) with pros and cons. Users can mix and match files across proposals in the WebUI. The turn's top-performing reference model harmonizes cross-file imports and contracts, followed by a final judge re-audit before promotion. Companion integration with [`dsh-time-machine`](https://github.com/goodandready/dsh-time-machine) for safe pre-promotion rollback checkpoints.
|
|
55
|
+
12. **Adversarial White-Hat Security Fuzzing Gate (#302)**: Red Team security audit scanning candidate code for SQLi, XSS, Command Injection, Path Traversal, ReDoS, and Prototype Pollution with blocking `SECURITY_VETO`.
|
|
56
|
+
13. **Chain-of-Verification (CoVe) Fact-Checking Gate for Code (#314)**: Independent pre-judge verification formulating factual API questions, validating modules and functions against standard library and project manifests to eliminate hallucinated methods.
|
|
55
57
|
|
|
56
58
|
---
|
|
57
59
|
|
package/README.ru.md
CHANGED
|
@@ -52,6 +52,8 @@
|
|
|
52
52
|
9. **История запусков и лидерборд моделей**: Персистентное логирование всех видов запусков (синтез, fast mode, опросник) и REST-эндпоинты (`/dsh-moa/history`, `/dsh-moa/leaderboard`, `/dsh-moa/runs/<id>`).
|
|
53
53
|
10. **Live Canvas 1-клик предпросмотр (опционально)**: если в профиле установлен `@goodandready/dsh-live-canvas`, промоученный HTML отправляется в его песочницу, а ответ MoA содержит ссылку на предпросмотр в 1 клик; без плагина шаг тихо пропускается.
|
|
54
54
|
11. **Интерактивный воркбенч (Оценка файлов 1–10, выбор в UI, склейка лучшей моделью и повторный аудит)**: Судья оценивает каждый сгенерированный файл каждого кандидата (1–10) с указанием ключевых плюсов и минусов. Пользователь комбинирует желаемый набор файлов в WebUI; лучшая модель раунда вызывается для гармонизации импортов и контрактов, после чего судья выполняет финальную валидацию перед промоушеном. Совместимо с [`dsh-time-machine`](https://github.com/goodandready/dsh-time-machine) для безопасных теневых снимков Git.
|
|
55
|
+
12. **Состязательный White-Hat фильтр безопасности (#302)**: Red Team аудит безопасности перед арбитражем судьи, выявляющий SQLi, XSS, инъекции команд, Path Traversal, ReDoS и Prototype Pollution с блокирующим вердиктом `SECURITY_VETO`.
|
|
56
|
+
13. **Фильтр верификации фактов и API (CoVe) (#314)**: Автономная цепочка верификации (Chain-of-Verification), формулирующая проверочные вопросы по вызовам методов и пакетам, исключающая галлюцинированные и несуществующие API библиотек.
|
|
55
57
|
|
|
56
58
|
---
|
|
57
59
|
|
package/README.zh.md
CHANGED
|
@@ -52,6 +52,8 @@
|
|
|
52
52
|
9. **运行历史与胜率排行榜**:对每一类运行(综合、快速模式、问卷)进行持久化记录,并内置 REST 端点(`/dsh-moa/history`、`/dsh-moa/leaderboard`、`/dsh-moa/runs/<id>`)。
|
|
53
53
|
10. **Live Canvas 一键预览(可选)**:当同一 profile 中安装了 `@goodandready/dsh-live-canvas` 时,提升到项目根目录的 HTML 会被推入其沙箱,MoA 回答附带一键预览链接;未安装时该步骤静默跳过。
|
|
54
54
|
11. **交互式工作台 (1-10分文件打分、自由混搭、最优模型粘合与裁判复审)**:裁判对每个候选模型生成的每个文件进行 1–10 分打分并列出优缺点。用户可在 WebUI 中自由勾选组合所需文件;当轮最高分模型被调用以粘合协同跨文件导入与类型接口,随后由裁判进行最终复审。支持配套插件 [`dsh-time-machine`](https://github.com/goodandready/dsh-time-machine) 在文件提升前创建影子 Git 回滚快照。
|
|
55
|
+
12. **对抗性白帽安全模糊测试关卡 (#302)**:红队安全审计器在裁判前检查 SQLi、XSS、命令注入、路径穿越、ReDoS 和原型污染漏洞,发现严重漏洞时触发 `SECURITY_VETO` 否决。
|
|
56
|
+
13. **代码事实与 API 验证链关卡 (CoVe) (#314)**:独立的裁判前事实核验,生成针对性 API 问题并对照标准库及项目配置清单,杜绝调用虚构不存在的幻觉方法。
|
|
55
57
|
|
|
56
58
|
---
|
|
57
59
|
|
package/lib/client.js
CHANGED
|
@@ -228,7 +228,11 @@ window.__ModuleLoader__.load({
|
|
|
228
228
|
"aggregator.stack_personas_title": "\u{1F4E6} Stack-Aware Dynamic Persona Evolution (Project Manifests)",
|
|
229
229
|
"aggregator.stack_personas_desc": "Scans project manifests (package.json, Cargo.toml, pyproject.toml, go.mod) and assigns specialized engineering roles to candidate slots.",
|
|
230
230
|
"aggregator.preferences_title": "\u{1F9E0} Developer Coding Principles & Preferences Metaprompt",
|
|
231
|
-
"aggregator.preferences_desc": "Injects learned developer preferences (strict typing, functional immutability, error boundaries) to guide Judge synthesis."
|
|
231
|
+
"aggregator.preferences_desc": "Injects learned developer preferences (strict typing, functional immutability, error boundaries) to guide Judge synthesis.",
|
|
232
|
+
"aggregator.security_gate_title": "\u{1F6E1}\uFE0F Adversarial White-Hat Security Fuzzing Gate",
|
|
233
|
+
"aggregator.security_gate_desc": "Red-team security auditor checks candidate proposals for SQLi, XSS, ReDoS, Command Injection and Path Traversal with blocking SECURITY_VETO.",
|
|
234
|
+
"aggregator.cove_gate_title": "\u{1F50D} Chain-of-Verification (CoVe) Fact-Checking Gate",
|
|
235
|
+
"aggregator.cove_gate_desc": "Formulates API existence & method signature verification questions, disqualifying candidates that use hallucinated methods."
|
|
232
236
|
};
|
|
233
237
|
const zh = {
|
|
234
238
|
title: "\u667A\u80FD\u4F53\u6DF7\u5408 (MoA)",
|
|
@@ -447,7 +451,11 @@ window.__ModuleLoader__.load({
|
|
|
447
451
|
"aggregator.stack_personas_title": "\u{1F4E6} \u57FA\u4E8E\u9879\u76EE\u914D\u7F6E\u6E05\u5355\u7684\u611F\u77E5\u89D2\u8272\u6F14\u5316 (Stack-Aware Personas)",
|
|
448
452
|
"aggregator.stack_personas_desc": "\u81EA\u52A8\u626B\u63CF\u9879\u76EE\u6E05\u5355 (package.json, Cargo.toml, pyproject.toml, go.mod) \u5E76\u4E3A\u5404\u5019\u9009\u69FD\u4F4D\u6307\u6D3E\u4E13\u5C5E\u4E13\u4E1A\u89D2\u8272\u3002",
|
|
449
453
|
"aggregator.preferences_title": "\u{1F9E0} \u81EA\u5B66\u4E60\u5F00\u53D1\u8005\u4EE3\u7801\u504F\u597D\u5143\u63D0\u793A\u8BCD (Developer Preferences)",
|
|
450
|
-
"aggregator.preferences_desc": "\u5C06\u5F00\u53D1\u8005\u504F\u597D\uFF08\u4E25\u683C\u7C7B\u578B\u3001\u51FD\u6570\u4E0D\u53EF\u53D8\u6027\u3001\u9632\u5FA1\u6027\u9519\u8BEF\u5904\u7406\u7B49\uFF09\u6CE8\u5165\u88C1\u5224\u7EFC\u5408\u8BC4\u5BA1\u9636\u6BB5\u3002"
|
|
454
|
+
"aggregator.preferences_desc": "\u5C06\u5F00\u53D1\u8005\u504F\u597D\uFF08\u4E25\u683C\u7C7B\u578B\u3001\u51FD\u6570\u4E0D\u53EF\u53D8\u6027\u3001\u9632\u5FA1\u6027\u9519\u8BEF\u5904\u7406\u7B49\uFF09\u6CE8\u5165\u88C1\u5224\u7EFC\u5408\u8BC4\u5BA1\u9636\u6BB5\u3002",
|
|
455
|
+
"aggregator.security_gate_title": "\u{1F6E1}\uFE0F \u5BF9\u6297\u6027\u767D\u5E3D\u5B89\u5168\u6A21\u7CCA\u6D4B\u8BD5\u5173\u5361",
|
|
456
|
+
"aggregator.security_gate_desc": "\u7EA2\u961F\u5B89\u5168\u5BA1\u8BA1\u5668\u68C0\u67E5\u5019\u9009\u65B9\u6848\u4E2D\u7684 SQLi\u3001XSS\u3001ReDoS\u3001\u547D\u4EE4\u6CE8\u5165\u548C\u8DEF\u5F84\u7A7F\u8D8A\u6F0F\u6D1E\uFF0C\u5E76\u5728\u53D1\u73B0\u4E25\u91CD\u6F0F\u6D1E\u65F6\u89E6\u53D1 SECURITY_VETO \u5426\u51B3\u3002",
|
|
457
|
+
"aggregator.cove_gate_title": "\u{1F50D} \u9A8C\u8BC1\u94FE (CoVe) \u4EE3\u7801\u4E8B\u5B9E\u4E0E API \u6838\u67E5\u5173\u5361",
|
|
458
|
+
"aggregator.cove_gate_desc": "\u751F\u6210 API \u5B58\u5728\u6027\u4E0E\u65B9\u6CD5\u7B7E\u540D\u6838\u9A8C\u95EE\u9898\uFF0C\u81EA\u52A8\u60E9\u7F5A\u6216\u53D6\u6D88\u4F7F\u7528\u865A\u6784\u5E7B\u89C9\u65B9\u6CD5\u7684\u5019\u9009\u65B9\u6848\u3002"
|
|
451
459
|
};
|
|
452
460
|
function makeT(dict, fallback) {
|
|
453
461
|
return function t(key, vars) {
|
|
@@ -2610,6 +2618,48 @@ window.__ModuleLoader__.load({
|
|
|
2610
2618
|
t("aggregator.preferences_desc") || "Injects learned developer preferences (strict typing, functional immutability, error boundaries) to guide Judge synthesis."
|
|
2611
2619
|
)
|
|
2612
2620
|
),
|
|
2621
|
+
/* Adversarial White-Hat Security Fuzzing Gate (#302) */
|
|
2622
|
+
React.createElement(
|
|
2623
|
+
"div",
|
|
2624
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6, padding: "8px 12px", background: "var(--dsw-alias-bg-layer-2)", borderRadius: 8, border: "1px solid var(--dsw-alias-border-l2)", marginTop: 8 } },
|
|
2625
|
+
React.createElement(
|
|
2626
|
+
"label",
|
|
2627
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, cursor: "pointer", fontSize: 13, fontWeight: 500 } },
|
|
2628
|
+
React.createElement("input", {
|
|
2629
|
+
type: "checkbox",
|
|
2630
|
+
checked: Boolean(currentPreset.security_gate_enabled),
|
|
2631
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, security_gate_enabled: e.target.checked })),
|
|
2632
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2633
|
+
}),
|
|
2634
|
+
t("aggregator.security_gate_title") || "\u{1F6E1}\uFE0F Adversarial White-Hat Security Fuzzing Gate"
|
|
2635
|
+
),
|
|
2636
|
+
React.createElement(
|
|
2637
|
+
"div",
|
|
2638
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2639
|
+
t("aggregator.security_gate_desc") || "Red-team security auditor checks candidate proposals for SQLi, XSS, ReDoS, Command Injection and Path Traversal with blocking SECURITY_VETO."
|
|
2640
|
+
)
|
|
2641
|
+
),
|
|
2642
|
+
/* Chain-of-Verification (CoVe) Fact-Checking Gate (#314) */
|
|
2643
|
+
React.createElement(
|
|
2644
|
+
"div",
|
|
2645
|
+
{ style: { display: "flex", flexDirection: "column", gap: 6, padding: "8px 12px", background: "var(--dsw-alias-bg-layer-2)", borderRadius: 8, border: "1px solid var(--dsw-alias-border-l2)", marginTop: 8 } },
|
|
2646
|
+
React.createElement(
|
|
2647
|
+
"label",
|
|
2648
|
+
{ style: { display: "flex", alignItems: "center", gap: 8, cursor: "pointer", fontSize: 13, fontWeight: 500 } },
|
|
2649
|
+
React.createElement("input", {
|
|
2650
|
+
type: "checkbox",
|
|
2651
|
+
checked: Boolean(currentPreset.cove_verification_enabled),
|
|
2652
|
+
onChange: (e) => updateCurrentPreset((p) => ({ ...p, cove_verification_enabled: e.target.checked })),
|
|
2653
|
+
style: { accentColor: "var(--dsw-alias-state-brand-primary, var(--dsw-alias-label-primary))", cursor: "pointer" }
|
|
2654
|
+
}),
|
|
2655
|
+
t("aggregator.cove_gate_title") || "\u{1F50D} Chain-of-Verification (CoVe) Fact-Checking Gate"
|
|
2656
|
+
),
|
|
2657
|
+
React.createElement(
|
|
2658
|
+
"div",
|
|
2659
|
+
{ style: { fontSize: 12, color: "var(--dsw-alias-label-secondary)", marginLeft: 22 } },
|
|
2660
|
+
t("aggregator.cove_gate_desc") || "Formulates API existence & method signature verification questions, disqualifying candidates that use hallucinated methods."
|
|
2661
|
+
)
|
|
2662
|
+
),
|
|
2613
2663
|
/* Fallback Judges Chain */
|
|
2614
2664
|
React.createElement(
|
|
2615
2665
|
"div",
|
package/lib/events.js
CHANGED
|
@@ -11,6 +11,8 @@ export const MOA_EVENTS = {
|
|
|
11
11
|
SELF_CONSISTENCY_EXPAND: 'moa/self-consistency-expand',
|
|
12
12
|
STACK_DETECTED: 'moa/stack-detected',
|
|
13
13
|
PREFERENCES_APPLIED: 'moa/preferences-applied',
|
|
14
|
+
SECURITY_AUDIT: 'moa/security-audit',
|
|
15
|
+
COVE_VERIFIED: 'moa/cove-verified',
|
|
14
16
|
CRITIQUE_COMPLETE: 'moa/critique-complete',
|
|
15
17
|
JUDGE_VERDICT: 'moa/judge-verdict',
|
|
16
18
|
CONTEXT_PRUNING: 'moa/context-pruning',
|
|
@@ -36,4 +38,3 @@ export function emitEvent(ctx, eventName, payload) {
|
|
|
36
38
|
/* best-effort event dispatch: listener exceptions must never abort pipeline */
|
|
37
39
|
}
|
|
38
40
|
}
|
|
39
|
-
|
package/lib/moa-schema.js
CHANGED
|
@@ -91,6 +91,10 @@ export const PresetSchema = z.object({
|
|
|
91
91
|
escalation_judge: ModelSlotSchema.default({ provider: '', model: '' }),
|
|
92
92
|
stack_aware_personas_enabled: z.boolean().default(false),
|
|
93
93
|
developer_preferences_enabled: z.boolean().default(false),
|
|
94
|
+
security_gate_enabled: z.boolean().default(false),
|
|
95
|
+
security_gate_strict_veto: z.boolean().default(true),
|
|
96
|
+
cove_verification_enabled: z.boolean().default(false),
|
|
97
|
+
cove_max_questions: z.number().default(5),
|
|
94
98
|
})
|
|
95
99
|
|
|
96
100
|
export const Config = z.object({
|
|
@@ -128,6 +132,10 @@ export const Config = z.object({
|
|
|
128
132
|
escalation_judge: ModelSlotSchema.default({ provider: '', model: '' }).volatile(),
|
|
129
133
|
stack_aware_personas_enabled: z.boolean().default(false).volatile(),
|
|
130
134
|
developer_preferences_enabled: z.boolean().default(false).volatile(),
|
|
135
|
+
security_gate_enabled: z.boolean().default(false).volatile(),
|
|
136
|
+
security_gate_strict_veto: z.boolean().default(true).volatile(),
|
|
137
|
+
cove_verification_enabled: z.boolean().default(false).volatile(),
|
|
138
|
+
cove_max_questions: z.number().default(5).volatile(),
|
|
131
139
|
smart_routing_model: ModelSlotSchema.default({ provider: '', model: '' }).volatile(),
|
|
132
140
|
prices: z.dict(PriceRow).default({}).volatile(),
|
|
133
141
|
presets: z.array(PresetSchema).default([
|
|
@@ -197,4 +205,3 @@ export function plainConfig(value) {
|
|
|
197
205
|
}
|
|
198
206
|
return value
|
|
199
207
|
}
|
|
200
|
-
|
|
@@ -0,0 +1,401 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* DeepSeek Harness Mixture of Agents (MoA) — Chain-of-Verification (CoVe) Fact-Checking Gate (#314)
|
|
3
|
+
*
|
|
4
|
+
* Pre-Judge verification gate that analyzes candidate proposals, formulates 3-5 factual API & syntax
|
|
5
|
+
* verification questions, audits them against standard libraries & project context, and flags
|
|
6
|
+
* hallucinated methods or nonexistent modules before arbitration.
|
|
7
|
+
*/
|
|
8
|
+
|
|
9
|
+
import { MOA_EVENTS, emitEvent } from '../events.js'
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* Standard Node.js built-in modules.
|
|
13
|
+
*/
|
|
14
|
+
export const NODE_STDLIB_MODULES = new Set([
|
|
15
|
+
'assert', 'async_hooks', 'buffer', 'child_process', 'cluster', 'console',
|
|
16
|
+
'constants', 'crypto', 'dgram', 'diagnostics_channel', 'dns', 'domain',
|
|
17
|
+
'events', 'fs', 'fs/promises', 'http', 'http2', 'https', 'inspector',
|
|
18
|
+
'module', 'net', 'os', 'path', 'path/posix', 'path/win32', 'perf_hooks',
|
|
19
|
+
'process', 'punycode', 'querystring', 'readline', 'repl', 'stream',
|
|
20
|
+
'stream/promises', 'stream/web', 'string_decoder', 'test', 'timers',
|
|
21
|
+
'timers/promises', 'tls', 'trace_events', 'tty', 'url', 'util',
|
|
22
|
+
'v8', 'vm', 'wasi', 'worker_threads', 'zlib',
|
|
23
|
+
])
|
|
24
|
+
|
|
25
|
+
/**
|
|
26
|
+
* Common hallucinated APIs and their real replacements.
|
|
27
|
+
*/
|
|
28
|
+
export const KNOWN_HALLUCINATIONS = [
|
|
29
|
+
{
|
|
30
|
+
regex: /fs\.(?:readAllText|readTextFile|readText|readAsString)\s*\(/i,
|
|
31
|
+
hallucination: 'fs.readAllText / fs.readTextFile',
|
|
32
|
+
correction: 'fs.readFileSync(path, "utf8") or fs.promises.readFile(path, "utf8")',
|
|
33
|
+
reason: 'Method does not exist in Node.js fs module (often hallucinated from C# .NET or Deno).',
|
|
34
|
+
},
|
|
35
|
+
{
|
|
36
|
+
regex: /fs\.(?:writeAllText|writeTextFile|writeText|writeAsString)\s*\(/i,
|
|
37
|
+
hallucination: 'fs.writeAllText / fs.writeTextFile',
|
|
38
|
+
correction: 'fs.writeFileSync(path, data, "utf8") or fs.promises.writeFile(path, data, "utf8")',
|
|
39
|
+
reason: 'Method does not exist in Node.js fs module.',
|
|
40
|
+
},
|
|
41
|
+
{
|
|
42
|
+
regex: /fs\.readFileAsync\s*\(/i,
|
|
43
|
+
hallucination: 'fs.readFileAsync',
|
|
44
|
+
correction: 'fs.promises.readFile(...) or util.promisify(fs.readFile)',
|
|
45
|
+
reason: 'Legacy Bluebird promise naming is not a native Node.js fs method.',
|
|
46
|
+
},
|
|
47
|
+
{
|
|
48
|
+
regex: /path\.(?:normalizeUrl|combine|joinPath)\s*\(/i,
|
|
49
|
+
hallucination: 'path.combine / path.normalizeUrl',
|
|
50
|
+
correction: 'path.join(...) or new URL(...)',
|
|
51
|
+
reason: 'Method does not exist in Node.js path module.',
|
|
52
|
+
},
|
|
53
|
+
{
|
|
54
|
+
regex: /crypto\.(?:generateRandomUuidString|randomString|generateUuid)\s*\(/i,
|
|
55
|
+
hallucination: 'crypto.generateRandomUuidString',
|
|
56
|
+
correction: 'crypto.randomUUID() or crypto.randomBytes(...)',
|
|
57
|
+
reason: 'Nonexistent crypto helper hallucinated by language models.',
|
|
58
|
+
},
|
|
59
|
+
{
|
|
60
|
+
regex: /JSON\.(?:toObject|fromObject|encode|decode)\s*\(/i,
|
|
61
|
+
hallucination: 'JSON.toObject / JSON.fromObject / JSON.encode',
|
|
62
|
+
correction: 'JSON.parse(...) or JSON.stringify(...)',
|
|
63
|
+
reason: 'Invalid JSON global methods hallucinated from other ecosystems.',
|
|
64
|
+
},
|
|
65
|
+
{
|
|
66
|
+
regex: /Promise\.(?:wait|sleep|delay)\s*\(/i,
|
|
67
|
+
hallucination: 'Promise.sleep / Promise.wait / Promise.delay',
|
|
68
|
+
correction: 'new Promise(resolve => setTimeout(resolve, ms)) or timers/promises setTimeout',
|
|
69
|
+
reason: 'Native JavaScript Promise does not feature sleep/delay directly on Promise constructor.',
|
|
70
|
+
},
|
|
71
|
+
]
|
|
72
|
+
|
|
73
|
+
/**
|
|
74
|
+
* Extracts candidate imports, requires, and method calls to formulate verification questions.
|
|
75
|
+
*
|
|
76
|
+
* @param {string} code
|
|
77
|
+
* @param {object} [options]
|
|
78
|
+
* @returns {Array<object>} Up to maxQuestions questions
|
|
79
|
+
*/
|
|
80
|
+
export function generateCoveQuestions(code, options = {}) {
|
|
81
|
+
const maxQuestions = Math.min(10, Math.max(1, options.maxQuestions || 5))
|
|
82
|
+
const questions = []
|
|
83
|
+
if (!code || typeof code !== 'string') return questions
|
|
84
|
+
|
|
85
|
+
// 1. Check imports/requires
|
|
86
|
+
const importRegex = /(?:import\s+(?:\{[^}]+\}|\*\s+as\s+\w+|\w+)\s+from\s+['"]([^'"]+)['"]|require\s*\(\s*['"]([^'"]+)['"]\s*\))/g
|
|
87
|
+
let match
|
|
88
|
+
const seenModules = new Set()
|
|
89
|
+
while ((match = importRegex.exec(code)) !== null) {
|
|
90
|
+
const mod = match[1] || match[2]
|
|
91
|
+
if (mod && !seenModules.has(mod) && !mod.startsWith('.') && !mod.startsWith('/')) {
|
|
92
|
+
seenModules.add(mod)
|
|
93
|
+
const cleanMod = mod.replace(/^node:/, '')
|
|
94
|
+
questions.push({
|
|
95
|
+
id: questions.length + 1,
|
|
96
|
+
type: 'module_resolution',
|
|
97
|
+
subject: mod,
|
|
98
|
+
cleanSubject: cleanMod,
|
|
99
|
+
question: `Is module '${mod}' available in standard library or project dependencies?`,
|
|
100
|
+
})
|
|
101
|
+
if (questions.length >= maxQuestions) return questions
|
|
102
|
+
}
|
|
103
|
+
}
|
|
104
|
+
|
|
105
|
+
// 2. Check for known hallucination method calls
|
|
106
|
+
for (const h of KNOWN_HALLUCINATIONS) {
|
|
107
|
+
if (h.regex.test(code)) {
|
|
108
|
+
questions.push({
|
|
109
|
+
id: questions.length + 1,
|
|
110
|
+
type: 'api_existence',
|
|
111
|
+
subject: h.hallucination,
|
|
112
|
+
hallucinationDef: h,
|
|
113
|
+
question: `Does API '${h.hallucination}' exist and is it valid in the target runtime?`,
|
|
114
|
+
})
|
|
115
|
+
if (questions.length >= maxQuestions) return questions
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
// 3. Check general API calls (fs, path, crypto, child_process)
|
|
120
|
+
const apiCallRegex = /\b(fs|path|crypto|os|child_process|JSON|Promise)\.([a-zA-Z0-9_]+)\s*\(/g
|
|
121
|
+
const seenApis = new Set()
|
|
122
|
+
while ((match = apiCallRegex.exec(code)) !== null) {
|
|
123
|
+
const obj = match[1]
|
|
124
|
+
const method = match[2]
|
|
125
|
+
const full = `${obj}.${method}`
|
|
126
|
+
if (!seenApis.has(full)) {
|
|
127
|
+
seenApis.add(full)
|
|
128
|
+
questions.push({
|
|
129
|
+
id: questions.length + 1,
|
|
130
|
+
type: 'method_signature',
|
|
131
|
+
subject: full,
|
|
132
|
+
object: obj,
|
|
133
|
+
method,
|
|
134
|
+
question: `Does method '${method}' natively exist on standard global '${obj}'?`,
|
|
135
|
+
})
|
|
136
|
+
if (questions.length >= maxQuestions) return questions
|
|
137
|
+
}
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
// Fallback question if code has no special calls
|
|
141
|
+
if (questions.length === 0) {
|
|
142
|
+
questions.push({
|
|
143
|
+
id: 1,
|
|
144
|
+
type: 'syntax_coherence',
|
|
145
|
+
subject: 'code_structure',
|
|
146
|
+
question: 'Are all functions and return types syntactically aligned and verifiable?',
|
|
147
|
+
})
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
return questions.slice(0, maxQuestions)
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
/**
|
|
154
|
+
* Verifies a single CoVe factual question against stdlib and project dependencies.
|
|
155
|
+
*
|
|
156
|
+
* @param {object} q - Question object from generateCoveQuestions
|
|
157
|
+
* @param {object} [projectContext]
|
|
158
|
+
* @returns {object} Verification result
|
|
159
|
+
*/
|
|
160
|
+
export function verifyCoveQuestion(q, projectContext = {}) {
|
|
161
|
+
const deps = {
|
|
162
|
+
...(projectContext.dependencies || {}),
|
|
163
|
+
...(projectContext.devDependencies || {}),
|
|
164
|
+
...(projectContext.stackInfo?.manifestDependencies || {}),
|
|
165
|
+
}
|
|
166
|
+
|
|
167
|
+
if (q.type === 'module_resolution') {
|
|
168
|
+
const isNode = NODE_STDLIB_MODULES.has(q.cleanSubject)
|
|
169
|
+
const isDep = Boolean(deps[q.cleanSubject] || deps[q.subject])
|
|
170
|
+
if (isNode || isDep) {
|
|
171
|
+
return {
|
|
172
|
+
verified: true,
|
|
173
|
+
confidence: 1.0,
|
|
174
|
+
rationale: isNode
|
|
175
|
+
? `Module '${q.subject}' is a verified Node.js standard library module.`
|
|
176
|
+
: `Module '${q.subject}' is defined in project manifest dependencies.`,
|
|
177
|
+
hallucination: null,
|
|
178
|
+
}
|
|
179
|
+
}
|
|
180
|
+
return {
|
|
181
|
+
verified: false,
|
|
182
|
+
confidence: 0.85,
|
|
183
|
+
rationale: `Module '${q.subject}' is not found in Node.js stdlib or declared project dependencies.`,
|
|
184
|
+
hallucination: {
|
|
185
|
+
api: q.subject,
|
|
186
|
+
reason: `Module '${q.subject}' was referenced without declaration in package.json or stdlib.`,
|
|
187
|
+
},
|
|
188
|
+
}
|
|
189
|
+
}
|
|
190
|
+
|
|
191
|
+
if (q.type === 'api_existence') {
|
|
192
|
+
// Matched a known hallucination pattern
|
|
193
|
+
const h = q.hallucinationDef
|
|
194
|
+
return {
|
|
195
|
+
verified: false,
|
|
196
|
+
confidence: 0.95,
|
|
197
|
+
rationale: `Invalid API detected: ${h.reason} Suggested fix: ${h.correction}`,
|
|
198
|
+
hallucination: {
|
|
199
|
+
api: h.hallucination,
|
|
200
|
+
reason: h.reason,
|
|
201
|
+
correction: h.correction,
|
|
202
|
+
},
|
|
203
|
+
}
|
|
204
|
+
}
|
|
205
|
+
|
|
206
|
+
if (q.type === 'method_signature') {
|
|
207
|
+
// Verify common valid methods on known objects
|
|
208
|
+
const validFs = new Set(['readFile', 'readFileSync', 'writeFile', 'writeFileSync', 'existsSync', 'statSync', 'stat', 'mkdirSync', 'mkdir', 'unlinkSync', 'unlink', 'readdirSync', 'readdir', 'createReadStream', 'createWriteStream', 'promises'])
|
|
209
|
+
const validPath = new Set(['join', 'resolve', 'basename', 'dirname', 'extname', 'relative', 'normalize', 'isAbsolute', 'parse', 'format', 'sep', 'posix', 'win32'])
|
|
210
|
+
const validCrypto = new Set(['randomUUID', 'randomBytes', 'createHash', 'createHmac', 'createCipheriv', 'createDecipheriv', 'timingSafeEqual', 'scryptSync', 'pbkdf2Sync'])
|
|
211
|
+
const validJson = new Set(['parse', 'stringify'])
|
|
212
|
+
const validPromise = new Set(['all', 'allSettled', 'race', 'any', 'resolve', 'reject'])
|
|
213
|
+
|
|
214
|
+
let isValid = true
|
|
215
|
+
if (q.object === 'fs') isValid = validFs.has(q.method)
|
|
216
|
+
else if (q.object === 'path') isValid = validPath.has(q.method)
|
|
217
|
+
else if (q.object === 'crypto') isValid = validCrypto.has(q.method)
|
|
218
|
+
else if (q.object === 'JSON') isValid = validJson.has(q.method)
|
|
219
|
+
else if (q.object === 'Promise') isValid = validPromise.has(q.method)
|
|
220
|
+
|
|
221
|
+
if (isValid) {
|
|
222
|
+
return {
|
|
223
|
+
verified: true,
|
|
224
|
+
confidence: 0.95,
|
|
225
|
+
rationale: `'${q.subject}' is a verified standard method.`,
|
|
226
|
+
hallucination: null,
|
|
227
|
+
}
|
|
228
|
+
}
|
|
229
|
+
return {
|
|
230
|
+
verified: false,
|
|
231
|
+
confidence: 0.9,
|
|
232
|
+
rationale: `'${q.method}' does not exist on global '${q.object}'.`,
|
|
233
|
+
hallucination: {
|
|
234
|
+
api: q.subject,
|
|
235
|
+
reason: `Method '${q.method}' does not exist on '${q.object}'.`,
|
|
236
|
+
},
|
|
237
|
+
}
|
|
238
|
+
}
|
|
239
|
+
|
|
240
|
+
return {
|
|
241
|
+
verified: true,
|
|
242
|
+
confidence: 0.9,
|
|
243
|
+
rationale: 'Verified general syntax structure.',
|
|
244
|
+
hallucination: null,
|
|
245
|
+
}
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
/**
|
|
249
|
+
* Audits a single candidate solution using Chain-of-Verification (CoVe).
|
|
250
|
+
*
|
|
251
|
+
* @param {string} code
|
|
252
|
+
* @param {object} [projectContext]
|
|
253
|
+
* @param {object} [options]
|
|
254
|
+
* @returns {object}
|
|
255
|
+
*/
|
|
256
|
+
export function auditCandidateCove(code, projectContext = {}, options = {}) {
|
|
257
|
+
const questions = generateCoveQuestions(code, options)
|
|
258
|
+
const verifications = []
|
|
259
|
+
const hallucinations = []
|
|
260
|
+
|
|
261
|
+
for (const q of questions) {
|
|
262
|
+
const v = verifyCoveQuestion(q, projectContext)
|
|
263
|
+
verifications.push({
|
|
264
|
+
...q,
|
|
265
|
+
verified: v.verified,
|
|
266
|
+
confidence: v.confidence,
|
|
267
|
+
rationale: v.rationale,
|
|
268
|
+
})
|
|
269
|
+
if (!v.verified && v.hallucination) {
|
|
270
|
+
hallucinations.push(v.hallucination)
|
|
271
|
+
}
|
|
272
|
+
}
|
|
273
|
+
|
|
274
|
+
const penalty = hallucinations.length * 35
|
|
275
|
+
const coveScore = Math.max(0, 100 - penalty)
|
|
276
|
+
const passed = hallucinations.length === 0
|
|
277
|
+
|
|
278
|
+
let summary = ''
|
|
279
|
+
if (passed) {
|
|
280
|
+
summary = `🔍 CoVe Fact-Checking: PASSED (Score 100/100). Verified ${questions.length} factual API/module question(s).`
|
|
281
|
+
} else {
|
|
282
|
+
summary = `⚠️ CoVe Hallucination Detected (Score ${coveScore}/100): ${hallucinations.length} non-existent or unverified API(s) found.`
|
|
283
|
+
}
|
|
284
|
+
|
|
285
|
+
return {
|
|
286
|
+
passed,
|
|
287
|
+
coveScore,
|
|
288
|
+
questions: verifications,
|
|
289
|
+
hallucinations,
|
|
290
|
+
summary,
|
|
291
|
+
}
|
|
292
|
+
}
|
|
293
|
+
|
|
294
|
+
/**
|
|
295
|
+
* Runs the Chain-of-Verification Gate across all candidates.
|
|
296
|
+
*
|
|
297
|
+
* @param {Array<object>} candidates
|
|
298
|
+
* @param {object} [projectContext]
|
|
299
|
+
* @param {object} [options]
|
|
300
|
+
* @returns {object}
|
|
301
|
+
*/
|
|
302
|
+
export function runCoveVerificationGate(candidates, projectContext = {}, options = {}) {
|
|
303
|
+
const { ctx, runId, onProgress } = options
|
|
304
|
+
const results = []
|
|
305
|
+
let totalHallucinations = 0
|
|
306
|
+
|
|
307
|
+
if (!Array.isArray(candidates) || candidates.length === 0) {
|
|
308
|
+
return {
|
|
309
|
+
results: [],
|
|
310
|
+
passedCount: 0,
|
|
311
|
+
hallucinationCount: 0,
|
|
312
|
+
coveDirective: null,
|
|
313
|
+
}
|
|
314
|
+
}
|
|
315
|
+
|
|
316
|
+
for (let i = 0; i < candidates.length; i++) {
|
|
317
|
+
const cand = candidates[i]
|
|
318
|
+
const text = typeof cand === 'string' ? cand : cand.output || cand.text || cand.code || ''
|
|
319
|
+
const report = auditCandidateCove(text, projectContext, options)
|
|
320
|
+
|
|
321
|
+
if (typeof cand === 'object' && cand !== null) {
|
|
322
|
+
cand.coveReport = report
|
|
323
|
+
cand.coveScore = report.coveScore
|
|
324
|
+
cand.coveHallucinations = report.hallucinations
|
|
325
|
+
}
|
|
326
|
+
|
|
327
|
+
totalHallucinations += report.hallucinations.length
|
|
328
|
+
results.push({
|
|
329
|
+
candidateIndex: i,
|
|
330
|
+
model: cand?.model || `Candidate ${i + 1}`,
|
|
331
|
+
report,
|
|
332
|
+
})
|
|
333
|
+
|
|
334
|
+
if (ctx) {
|
|
335
|
+
emitEvent(ctx, MOA_EVENTS.COVE_VERIFIED, {
|
|
336
|
+
runId,
|
|
337
|
+
candidateIndex: i,
|
|
338
|
+
coveScore: report.coveScore,
|
|
339
|
+
passed: report.passed,
|
|
340
|
+
questionsCount: report.questions.length,
|
|
341
|
+
hallucinationsCount: report.hallucinations.length,
|
|
342
|
+
})
|
|
343
|
+
}
|
|
344
|
+
}
|
|
345
|
+
|
|
346
|
+
const passedCount = results.filter((r) => r.report.passed).length
|
|
347
|
+
|
|
348
|
+
if (typeof onProgress === 'function') {
|
|
349
|
+
if (totalHallucinations === 0) {
|
|
350
|
+
onProgress(`🔍 *CoVe Fact-Check Passed*: All APIs and modules verified against project context.\n`)
|
|
351
|
+
} else {
|
|
352
|
+
onProgress(`⚠️ *CoVe Fact-Check Alert*: Found ${totalHallucinations} unverified or hallucinated API(s) across candidate solutions.\n`)
|
|
353
|
+
}
|
|
354
|
+
}
|
|
355
|
+
|
|
356
|
+
const coveDirective = formatCoveDirective(results)
|
|
357
|
+
|
|
358
|
+
return {
|
|
359
|
+
results,
|
|
360
|
+
passedCount,
|
|
361
|
+
hallucinationCount: totalHallucinations,
|
|
362
|
+
coveDirective,
|
|
363
|
+
}
|
|
364
|
+
}
|
|
365
|
+
|
|
366
|
+
/**
|
|
367
|
+
* Formats CoVe fact-checking verification findings into a prompt directive for Judge/Synthesis.
|
|
368
|
+
*
|
|
369
|
+
* @param {Array<object>} results
|
|
370
|
+
* @returns {string|null}
|
|
371
|
+
*/
|
|
372
|
+
export function formatCoveDirective(results) {
|
|
373
|
+
if (!Array.isArray(results) || results.length === 0) return null
|
|
374
|
+
|
|
375
|
+
const hasAnyHallucinations = results.some((r) => r.report && r.report.hallucinations.length > 0)
|
|
376
|
+
if (!hasAnyHallucinations) {
|
|
377
|
+
return '### 🔍 CHAIN-OF-VERIFICATION (CoVe): ALL CANDIDATE APIS VERIFIED FACTUAL\nNo hallucinated APIs, phantom modules, or broken native signatures detected.'
|
|
378
|
+
}
|
|
379
|
+
|
|
380
|
+
const lines = [
|
|
381
|
+
'### 🔍 CHAIN-OF-VERIFICATION (CoVe) FACT-CHECK DIRECTIVE',
|
|
382
|
+
'Independent factual audit of API calls and module imports has been completed:',
|
|
383
|
+
]
|
|
384
|
+
|
|
385
|
+
for (const r of results) {
|
|
386
|
+
const label = `Candidate ${r.candidateIndex + 1}`
|
|
387
|
+
const rep = r.report || {}
|
|
388
|
+
if (!rep.passed && rep.hallucinations.length > 0) {
|
|
389
|
+
lines.push(`- **${label}**: ❌ **FACT-CHECK FAILURE** (Score ${rep.coveScore}/100) — Hallucinated APIs detected:`)
|
|
390
|
+
for (const h of rep.hallucinations) {
|
|
391
|
+
lines.push(` * \`${h.api}\`: ${h.reason}${h.correction ? ` (Correction: ${h.correction})` : ''}`)
|
|
392
|
+
}
|
|
393
|
+
} else {
|
|
394
|
+
lines.push(`- **${label}**: ✅ VERIFIED (Score 100/100) — All ${rep.questions?.length || 0} API checks confirmed valid.`)
|
|
395
|
+
}
|
|
396
|
+
}
|
|
397
|
+
|
|
398
|
+
lines.push('\n**Judge Fact-Checking Rule**: Do not award winning status to solutions that rely on hallucinated methods or nonexistent packages. Prefer solutions with 100/100 verified API accuracy.')
|
|
399
|
+
|
|
400
|
+
return lines.join('\n')
|
|
401
|
+
}
|
package/lib/pipeline/runner.js
CHANGED
|
@@ -18,6 +18,8 @@ import { stageSynthesis } from './synthesis.js'
|
|
|
18
18
|
import { stagePromotion } from './promotion.js'
|
|
19
19
|
import { applySyntaxFilter, formatSyntaxDirective } from './syntax-filter.js'
|
|
20
20
|
import { stageEarlyConsensus } from './early-consensus.js'
|
|
21
|
+
import { runSecurityAuditGate } from './security-fuzzer.js'
|
|
22
|
+
import { runCoveVerificationGate } from './cove-verifier.js'
|
|
21
23
|
import { emitEvent, MOA_EVENTS } from '../events.js'
|
|
22
24
|
|
|
23
25
|
export async function runMoAPipeline({
|
|
@@ -199,6 +201,39 @@ export async function runMoAPipeline({
|
|
|
199
201
|
return { error: new Error('Turn aborted') }
|
|
200
202
|
}
|
|
201
203
|
|
|
204
|
+
// Pre-Stage 5.8: Chain-of-Verification (CoVe) Fact-Checking Gate (#314)
|
|
205
|
+
let coveDirective = null
|
|
206
|
+
let coveGateResult = null
|
|
207
|
+
if (preset?.cove_verification_enabled) {
|
|
208
|
+
coveGateResult = runCoveVerificationGate(
|
|
209
|
+
syntaxResult.validCandidates.length > 0 ? syntaxResult.validCandidates : successfulRefs,
|
|
210
|
+
ctxStage,
|
|
211
|
+
{
|
|
212
|
+
ctx,
|
|
213
|
+
runId,
|
|
214
|
+
onProgress,
|
|
215
|
+
maxQuestions: preset.cove_max_questions || 5,
|
|
216
|
+
}
|
|
217
|
+
)
|
|
218
|
+
coveDirective = coveGateResult.coveDirective
|
|
219
|
+
}
|
|
220
|
+
|
|
221
|
+
// Pre-Stage 5.9: Adversarial White-Hat Security Fuzzing Gate (#302)
|
|
222
|
+
let securityDirective = null
|
|
223
|
+
let securityGateResult = null
|
|
224
|
+
if (preset?.security_gate_enabled) {
|
|
225
|
+
securityGateResult = runSecurityAuditGate(
|
|
226
|
+
syntaxResult.validCandidates.length > 0 ? syntaxResult.validCandidates : successfulRefs,
|
|
227
|
+
{
|
|
228
|
+
ctx,
|
|
229
|
+
runId,
|
|
230
|
+
onProgress,
|
|
231
|
+
strictVeto: preset.security_gate_strict_veto !== false,
|
|
232
|
+
}
|
|
233
|
+
)
|
|
234
|
+
securityDirective = securityGateResult.securityDirective
|
|
235
|
+
}
|
|
236
|
+
|
|
202
237
|
// Pre-Stage 6: Dynamic Consensus Threshold & Multi-Judge Escalation (#300)
|
|
203
238
|
const escalationResult = evaluateJudgeEscalation({
|
|
204
239
|
candidateOutputs: syntaxResult.validCandidates.length > 0 ? syntaxResult.validCandidates : successfulRefs,
|
|
@@ -249,7 +284,7 @@ export async function runMoAPipeline({
|
|
|
249
284
|
preset: effectivePreset,
|
|
250
285
|
successfulRefs: syntaxResult.validCandidates.length > 0 ? syntaxResult.validCandidates : successfulRefs,
|
|
251
286
|
referenceOutputs, primaryJudge: effectivePrimaryJudge,
|
|
252
|
-
userPrompt, judgeCriteria: ctxStage.judgeCriteria, prices, maxTokens: ctxStage.maxTokens,
|
|
287
|
+
userPrompt, judgeCriteria: [ctxStage.judgeCriteria, coveDirective, securityDirective].filter(Boolean).join('\n\n'), prices, maxTokens: ctxStage.maxTokens,
|
|
253
288
|
callLlm, signal, onProgress, cwd, runId, isRefinement: ctxStage.isRefinement,
|
|
254
289
|
isFastMode: ctxStage.isFastMode, collectedCtx: ctxStage.collectedCtx, startTime,
|
|
255
290
|
})
|
|
@@ -307,7 +342,7 @@ export async function runMoAPipeline({
|
|
|
307
342
|
}
|
|
308
343
|
}
|
|
309
344
|
|
|
310
|
-
const fileScoringDirective = [rawFileDirective, visualResult?.directive, adversarialTddResult?.directive, syntaxDirective, pruningDirective, ctxStage.stackAwareDirective, ctxStage.developerPreferencesDirective].filter(Boolean).join("\n\n") || null
|
|
345
|
+
const fileScoringDirective = [rawFileDirective, visualResult?.directive, adversarialTddResult?.directive, syntaxDirective, coveDirective, securityDirective, pruningDirective, ctxStage.stackAwareDirective, ctxStage.developerPreferencesDirective].filter(Boolean).join("\n\n") || null
|
|
311
346
|
|
|
312
347
|
// Stage 7: Synthesis Phase
|
|
313
348
|
const synthRes = await stageSynthesis({
|
|
@@ -0,0 +1,290 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* DeepSeek Harness Mixture of Agents (MoA) — Adversarial Security Fuzzing Gate (#302)
|
|
3
|
+
*
|
|
4
|
+
* White-Hat Red Team security gate that audits candidate code proposals for critical
|
|
5
|
+
* vulnerabilities, injection vectors, memory exhaustion, prototype pollution, and CWE patterns
|
|
6
|
+
* before final Judge arbitration.
|
|
7
|
+
*/
|
|
8
|
+
|
|
9
|
+
import { MOA_EVENTS, emitEvent } from '../events.js'
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* Known vulnerability rule signatures with CWE classifications.
|
|
13
|
+
*/
|
|
14
|
+
export const SECURITY_RULES = [
|
|
15
|
+
{
|
|
16
|
+
cwe: 'CWE-89',
|
|
17
|
+
title: 'SQL Injection Risk',
|
|
18
|
+
severity: 'critical',
|
|
19
|
+
regex: /(?:SELECT|INSERT|UPDATE|DELETE|DROP|ALTER|CREATE|EXEC)\s+[^;`'"]*?(\$\{[^}]+\}|['"`]\s*\+\s*[a-zA-Z0-9_]+)/i,
|
|
20
|
+
detail: 'Unsanitized query construction using string interpolation or concatenation instead of parameterized bindings.',
|
|
21
|
+
},
|
|
22
|
+
{
|
|
23
|
+
cwe: 'CWE-78',
|
|
24
|
+
title: 'OS Command Injection',
|
|
25
|
+
severity: 'critical',
|
|
26
|
+
regex: /(?:child_process\.)?(?:exec|execSync)\s*\(\s*(?:`[^`]*\$\{[^}]+\}[^`]*`|['"][^'"]*['"]\s*\+)/i,
|
|
27
|
+
detail: 'Executing shell commands with interpolated or concatenated untrusted input.',
|
|
28
|
+
},
|
|
29
|
+
{
|
|
30
|
+
cwe: 'CWE-78',
|
|
31
|
+
title: 'Shell Invocation with Dynamic Arguments',
|
|
32
|
+
severity: 'critical',
|
|
33
|
+
regex: /spawn\s*\([^,]+,\s*\{[^}]*shell\s*:\s*true/i,
|
|
34
|
+
detail: 'Invoking child_process.spawn with shell: true exposes process to command injection.',
|
|
35
|
+
},
|
|
36
|
+
{
|
|
37
|
+
cwe: 'CWE-22',
|
|
38
|
+
title: 'Path Traversal Vulnerability',
|
|
39
|
+
severity: 'high',
|
|
40
|
+
regex: /(?:readFile|writeFile|createReadStream|createWriteStream|unlink|stat)\s*\([^)]*?(?:\.\.\/|\.\.\\)/i,
|
|
41
|
+
detail: 'File system operation contains unconstrained relative path traversal sequences (../).',
|
|
42
|
+
},
|
|
43
|
+
{
|
|
44
|
+
cwe: 'CWE-79',
|
|
45
|
+
title: 'Cross-Site Scripting (DOM XSS)',
|
|
46
|
+
severity: 'high',
|
|
47
|
+
regex: /(?:\.innerHTML\s*=|document\.write\s*\(|dangerouslySetInnerHTML\s*=\s*\{\s*\{\s*__html\s*:\s*(?!DOMPurify|sanitize))/i,
|
|
48
|
+
detail: 'Unescaped HTML or DOM insertion allows arbitrary script execution in browser contexts.',
|
|
49
|
+
},
|
|
50
|
+
{
|
|
51
|
+
cwe: 'CWE-1321',
|
|
52
|
+
title: 'Prototype Pollution',
|
|
53
|
+
severity: 'critical',
|
|
54
|
+
regex: /(?:__proto__|constructor\s*\[\s*['"`]prototype['"`]\s*\]|prototype\s*\[\s*['"`]\w+['"`]\s*\]\s*=)/i,
|
|
55
|
+
detail: 'Direct modification of object prototype chain may lead to remote code execution or privilege escalation.',
|
|
56
|
+
},
|
|
57
|
+
{
|
|
58
|
+
cwe: 'CWE-400',
|
|
59
|
+
title: 'Regular Expression Denial of Service (ReDoS)',
|
|
60
|
+
severity: 'high',
|
|
61
|
+
regex: /\/(?:[^\/\\]|\\.)*(?:\([^\)]*[\+\*]\)[^\)]*[\+\*]|\([^\)]*\|[^\)]*\)[\+\*])(?:[^\/\\]|\\.)*\//,
|
|
62
|
+
detail: 'Regular expression pattern contains nested quantifiers with catastrophic polynomial/exponential backtracking.',
|
|
63
|
+
},
|
|
64
|
+
{
|
|
65
|
+
cwe: 'CWE-798',
|
|
66
|
+
title: 'Hardcoded Cryptographic Secret / Credential',
|
|
67
|
+
severity: 'high',
|
|
68
|
+
regex: /(?:api_?key|auth_?token|secret_?key|private_?key|bearer_?token)\s*[:=]\s*['"`](?:sk_live_[0-9a-zA-Z]{16,}|ghp_[0-9a-zA-Z]{20,}|ey[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{20,})['"`]/i,
|
|
69
|
+
detail: 'Production-like API token, private key, or credential embedded directly in source code.',
|
|
70
|
+
},
|
|
71
|
+
{
|
|
72
|
+
cwe: 'CWE-362',
|
|
73
|
+
title: 'TOCTOU Concurrency Race Condition',
|
|
74
|
+
severity: 'medium',
|
|
75
|
+
regex: /fs\.existsSync\s*\([^)]+\)[\s\S]{1,80}?fs\.(?:writeFileSync|appendFileSync)\s*\(/i,
|
|
76
|
+
detail: 'Non-atomic check-then-act pattern creates file race condition under concurrent operations.',
|
|
77
|
+
},
|
|
78
|
+
]
|
|
79
|
+
|
|
80
|
+
/**
|
|
81
|
+
* Audits a code snippet or model response for security vulnerabilities.
|
|
82
|
+
*
|
|
83
|
+
* @param {string} code - Candidate response text or extracted code
|
|
84
|
+
* @param {object} [options]
|
|
85
|
+
* @returns {object} Audit report with score, findings, and veto flag
|
|
86
|
+
*/
|
|
87
|
+
export function auditCodeSecurity(code, options = {}) {
|
|
88
|
+
if (!code || typeof code !== 'string') {
|
|
89
|
+
return {
|
|
90
|
+
veto: false,
|
|
91
|
+
score: 100,
|
|
92
|
+
findings: [],
|
|
93
|
+
summary: 'Clean: No executable code provided for security audit.',
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
const findings = []
|
|
98
|
+
const lines = code.split('\n')
|
|
99
|
+
|
|
100
|
+
for (const rule of SECURITY_RULES) {
|
|
101
|
+
if (rule.regex.test(code)) {
|
|
102
|
+
// Pinpoint line number if possible
|
|
103
|
+
let matchedLine = null
|
|
104
|
+
let matchedSnippet = ''
|
|
105
|
+
for (let i = 0; i < lines.length; i++) {
|
|
106
|
+
if (rule.regex.test(lines[i])) {
|
|
107
|
+
matchedLine = i + 1
|
|
108
|
+
matchedSnippet = lines[i].trim().slice(0, 120)
|
|
109
|
+
break
|
|
110
|
+
}
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
findings.push({
|
|
114
|
+
cwe: rule.cwe,
|
|
115
|
+
title: rule.title,
|
|
116
|
+
severity: rule.severity,
|
|
117
|
+
detail: rule.detail,
|
|
118
|
+
line: matchedLine,
|
|
119
|
+
snippet: matchedSnippet || null,
|
|
120
|
+
})
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
let penalty = 0
|
|
125
|
+
let hasCriticalOrHigh = false
|
|
126
|
+
for (const f of findings) {
|
|
127
|
+
if (f.severity === 'critical') {
|
|
128
|
+
penalty += 45
|
|
129
|
+
hasCriticalOrHigh = true
|
|
130
|
+
} else if (f.severity === 'high') {
|
|
131
|
+
penalty += 25
|
|
132
|
+
hasCriticalOrHigh = true
|
|
133
|
+
} else if (f.severity === 'medium') {
|
|
134
|
+
penalty += 15
|
|
135
|
+
} else {
|
|
136
|
+
penalty += 5
|
|
137
|
+
}
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
const score = Math.max(0, Math.min(100, 100 - penalty))
|
|
141
|
+
const strictVeto = options.strictVeto !== false
|
|
142
|
+
const veto = strictVeto ? hasCriticalOrHigh || score < 60 : score < 40
|
|
143
|
+
|
|
144
|
+
let summary = ''
|
|
145
|
+
if (findings.length === 0) {
|
|
146
|
+
summary = '🛡️ Red Team Security Audit: PASSED (Score 100/100). No known CWE vulnerabilities detected.'
|
|
147
|
+
} else if (veto) {
|
|
148
|
+
summary = `⛔ SECURITY_VETO (Score ${score}/100): Found ${findings.length} security violation(s) (${findings.map((f) => f.cwe).join(', ')}).`
|
|
149
|
+
} else {
|
|
150
|
+
summary = `⚠️ Security Warning (Score ${score}/100): ${findings.length} minor/medium issue(s) identified.`
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
return {
|
|
154
|
+
veto,
|
|
155
|
+
score,
|
|
156
|
+
findings,
|
|
157
|
+
summary,
|
|
158
|
+
}
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
/**
|
|
162
|
+
* Executes the Adversarial Security Fuzzing Gate across all candidate outputs.
|
|
163
|
+
*
|
|
164
|
+
* @param {Array<object>} candidates - List of candidate references with output/text
|
|
165
|
+
* @param {object} [options]
|
|
166
|
+
* @returns {object} Gate execution results with directive and veto counters
|
|
167
|
+
*/
|
|
168
|
+
export function runSecurityAuditGate(candidates, options = {}) {
|
|
169
|
+
const { ctx, runId, onProgress, strictVeto = true } = options
|
|
170
|
+
const results = []
|
|
171
|
+
let vetoCount = 0
|
|
172
|
+
|
|
173
|
+
if (!Array.isArray(candidates) || candidates.length === 0) {
|
|
174
|
+
return {
|
|
175
|
+
results: [],
|
|
176
|
+
vetoCount: 0,
|
|
177
|
+
cleanCount: 0,
|
|
178
|
+
allPassed: true,
|
|
179
|
+
securityDirective: null,
|
|
180
|
+
}
|
|
181
|
+
}
|
|
182
|
+
|
|
183
|
+
for (let i = 0; i < candidates.length; i++) {
|
|
184
|
+
const cand = candidates[i]
|
|
185
|
+
const text = typeof cand === 'string' ? cand : cand.output || cand.text || cand.code || ''
|
|
186
|
+
const audit = auditCodeSecurity(text, { strictVeto })
|
|
187
|
+
|
|
188
|
+
if (typeof cand === 'object' && cand !== null) {
|
|
189
|
+
cand.securityAudit = audit
|
|
190
|
+
cand.securityVeto = audit.veto
|
|
191
|
+
}
|
|
192
|
+
|
|
193
|
+
if (audit.veto) {
|
|
194
|
+
vetoCount++
|
|
195
|
+
}
|
|
196
|
+
|
|
197
|
+
results.push({
|
|
198
|
+
candidateIndex: i,
|
|
199
|
+
model: cand?.model || `Candidate ${i + 1}`,
|
|
200
|
+
audit,
|
|
201
|
+
veto: audit.veto,
|
|
202
|
+
})
|
|
203
|
+
|
|
204
|
+
if (ctx) {
|
|
205
|
+
emitEvent(ctx, MOA_EVENTS.SECURITY_AUDIT, {
|
|
206
|
+
runId,
|
|
207
|
+
candidateIndex: i,
|
|
208
|
+
score: audit.score,
|
|
209
|
+
veto: audit.veto,
|
|
210
|
+
findingsCount: audit.findings.length,
|
|
211
|
+
cwes: audit.findings.map((f) => f.cwe),
|
|
212
|
+
})
|
|
213
|
+
}
|
|
214
|
+
}
|
|
215
|
+
|
|
216
|
+
const cleanCount = candidates.length - vetoCount
|
|
217
|
+
const allPassed = vetoCount === 0
|
|
218
|
+
|
|
219
|
+
if (typeof onProgress === 'function') {
|
|
220
|
+
if (allPassed) {
|
|
221
|
+
onProgress(`🛡️ *Security Gate Passed*: All ${candidates.length} candidate solutions cleared Red Team audit.\n`)
|
|
222
|
+
} else {
|
|
223
|
+
onProgress(
|
|
224
|
+
`🚨 *Security Gate Alert*: ${vetoCount}/${candidates.length} candidates triggered SECURITY_VETO (${results
|
|
225
|
+
.filter((r) => r.veto)
|
|
226
|
+
.map((r) => `Ref ${r.candidateIndex + 1}`)
|
|
227
|
+
.join(', ')}).\n`
|
|
228
|
+
)
|
|
229
|
+
}
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
const securityDirective = formatSecurityAuditDirective(results, { strictVeto })
|
|
233
|
+
|
|
234
|
+
return {
|
|
235
|
+
results,
|
|
236
|
+
vetoCount,
|
|
237
|
+
cleanCount,
|
|
238
|
+
allPassed,
|
|
239
|
+
securityDirective,
|
|
240
|
+
}
|
|
241
|
+
}
|
|
242
|
+
|
|
243
|
+
/**
|
|
244
|
+
* Formats Red Team audit findings into a high-priority prompt directive for Stage 6/7.
|
|
245
|
+
*
|
|
246
|
+
* @param {Array<object>} results
|
|
247
|
+
* @param {object} [options]
|
|
248
|
+
* @returns {string|null} Directive text
|
|
249
|
+
*/
|
|
250
|
+
export function formatSecurityAuditDirective(results, options = {}) {
|
|
251
|
+
if (!Array.isArray(results) || results.length === 0) return null
|
|
252
|
+
|
|
253
|
+
const hasAnyFindings = results.some((r) => r.audit && r.audit.findings.length > 0)
|
|
254
|
+
if (!hasAnyFindings) {
|
|
255
|
+
return '### 🛡️ RED TEAM SECURITY AUDIT: ALL CANDIDATES VERIFIED SECURE\nNo injection, XSS, ReDoS, path traversal, or credential exposure detected.'
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
const lines = [
|
|
259
|
+
'### 🛡️ ADVERSARIAL RED TEAM SECURITY AUDIT DIRECTIVE',
|
|
260
|
+
'Strict security evaluation of candidate proposals has been completed:',
|
|
261
|
+
]
|
|
262
|
+
|
|
263
|
+
for (const r of results) {
|
|
264
|
+
const label = `Candidate ${r.candidateIndex + 1}`
|
|
265
|
+
const a = r.audit || {}
|
|
266
|
+
if (a.veto) {
|
|
267
|
+
lines.push(
|
|
268
|
+
`- **${label}**: ⛔ **SECURITY_VETO** (Score ${a.score}/100) — DO NOT adopt this solution without removing critical flaws:`
|
|
269
|
+
)
|
|
270
|
+
for (const f of a.findings || []) {
|
|
271
|
+
lines.push(
|
|
272
|
+
` * [${f.cwe}] ${f.title} (${f.severity.toUpperCase()}): ${f.detail}${f.line ? ` (at line ${f.line})` : ''}`
|
|
273
|
+
)
|
|
274
|
+
}
|
|
275
|
+
} else if (a.findings && a.findings.length > 0) {
|
|
276
|
+
lines.push(`- **${label}**: ⚠️ Warning (Score ${a.score}/100):`)
|
|
277
|
+
for (const f of a.findings) {
|
|
278
|
+
lines.push(` * [${f.cwe}] ${f.title}: ${f.detail}`)
|
|
279
|
+
}
|
|
280
|
+
} else {
|
|
281
|
+
lines.push(`- **${label}**: ✅ PASSED (Score 100/100) — Free of detectable vulnerabilities.`)
|
|
282
|
+
}
|
|
283
|
+
}
|
|
284
|
+
|
|
285
|
+
lines.push(
|
|
286
|
+
'\n**Judge Mandate**: Solutions marked with SECURITY_VETO must be heavily penalized or rejected. The final synthesized code MUST NOT contain any of the flagged CWE patterns.'
|
|
287
|
+
)
|
|
288
|
+
|
|
289
|
+
return lines.join('\n')
|
|
290
|
+
}
|