@shubh000/macrc 0.1.0 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +38 -6
- package/cli/lib/{ast-verifier.mjs → context-verifier.mjs} +4 -4
- package/cli/lib/heuristic.mjs +47 -2
- package/cli/lib/syntax-queries.mjs +169 -0
- package/package.json +7 -2
- package/tests/{ast-verifier.test.mjs → context-verifier.test.mjs} +3 -3
- package/tests/fixtures/diff-after.json +36 -0
- package/tests/fixtures/diff-before.json +36 -0
- package/tests/integration.test.mjs +82 -0
- package/tests/syntax-queries.test.mjs +98 -0
package/README.md
CHANGED
|
@@ -143,6 +143,25 @@ Compares two audit JSON files and reports regressions and improvements.
|
|
|
143
143
|
|
|
144
144
|
---
|
|
145
145
|
|
|
146
|
+
## GitHub Code Scanning
|
|
147
|
+
|
|
148
|
+
MACRC outputs SARIF, which GitHub Code Scanning ingests natively:
|
|
149
|
+
|
|
150
|
+
```yaml
|
|
151
|
+
- name: MACRC audit
|
|
152
|
+
run: npx @shubh000/macrc gate . --format sarif --out macrc.sarif || true
|
|
153
|
+
|
|
154
|
+
- name: Upload MACRC results
|
|
155
|
+
uses: github/codeql-action/upload-sarif@v3
|
|
156
|
+
with:
|
|
157
|
+
sarif_file: macrc.sarif
|
|
158
|
+
category: macrc
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
Results appear in the **Security → Code scanning** tab. Use `|| true` if you want findings without blocking the workflow, or drop it to block on absent Tier-1 dims.
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
|
|
146
165
|
## GitHub Action
|
|
147
166
|
|
|
148
167
|
```yaml
|
|
@@ -162,7 +181,7 @@ Full inputs: `directory`, `format`, `policy-dir`, `stack-category`, `llm`, `sari
|
|
|
162
181
|
|
|
163
182
|
**Step 1: Heuristic pass (all 20 dims, deterministic)**
|
|
164
183
|
|
|
165
|
-
Three signal extractors run per repo. For
|
|
184
|
+
Three signal extractors run per repo. For 10 patterns with high false-positive risk (e.g. `zod_usage`, `bearer_auth`, `health_route`), hits go through a context-window pass: ±10 surrounding lines are checked with rule-based string analysis to confirm the signal is at a real call site, not a schema definition, test fixture, or comment. This is not an AST parser — no tree-sitter or Babel bindings. It is a fast, dependency-free signal verifier (`context-verifier.mjs`).
|
|
166
185
|
|
|
167
186
|
| Extractor | What it checks |
|
|
168
187
|
|---|---|
|
|
@@ -172,11 +191,17 @@ Three signal extractors run per repo. For the 10 patterns with high false-positi
|
|
|
172
191
|
|
|
173
192
|
**Step 2: Targeted LLM pass (Tier-1 dims only, `--llm` flag)**
|
|
174
193
|
|
|
175
|
-
|
|
194
|
+
5 dimensions have ≥20pp divergence between heuristic and LLM scores on the 232-repo corpus (231 used in LLM calibration — 1 quarantined during ingest). For these, a structured prompt is sent to `claude-haiku-4-5` with the actual route/controller code as context, only when the heuristic says `partial` or `present`.
|
|
195
|
+
|
|
196
|
+
**Why verify `partial/present`, not `absent`?** The Tier-1 gap is a *false-negative* gap: the heuristic says a repo has `auth` or `input_validation` (partial or present) when the LLM finds it actually doesn't. The heuristic almost never calls a truly present dim absent — it errs on the side of optimism. So the false negatives sit entirely in the `partial/present` bucket. Sending the LLM only those repos cuts the verification cost by ~70% (roughly 30% of repos per Tier-1 dim have partial/present) while catching the cases that matter. Verifying `absent` repos would add cost with zero TPR gain.
|
|
176
197
|
|
|
177
198
|
**Step 3: Patch generation (`fix` subcommand)**
|
|
178
199
|
|
|
179
|
-
For each absent/partial Tier-1 dim, a language-aware patch builder
|
|
200
|
+
For each absent/partial Tier-1 dim, a language-aware patch builder generates a minimal scaffolding patch: injecting middleware stubs after the last import block, adding timeout config to HTTP client calls, replacing hardcoded `localhost` URLs with env var expressions, wiring a Redis session store.
|
|
201
|
+
|
|
202
|
+
**Scope note:** patches target straightforward Express/FastAPI apps. They are starter scaffolding, not universal codemods — complex TypeScript generics, custom axios wrappers, or monorepo structures will need manual adjustment after apply.
|
|
203
|
+
|
|
204
|
+
**AST query roadmap:** Current acorn queries cover JS/TS only. Python equivalent via `python3 -m ast` subprocess (stdlib, no install) is the planned next addition — relevant for the AI/ML backend ecosystem where most AI-native services are FastAPI or Flask.
|
|
180
205
|
|
|
181
206
|
---
|
|
182
207
|
|
|
@@ -200,7 +225,7 @@ Gap = LLM absent% minus verified heuristic absent%. Tier-1 (≥20pp gap) means t
|
|
|
200
225
|
|
|
201
226
|
Contextual verification closes large parts of the gap for health_checks (+56pp raw vs +3pp verified) and load_balancing_awareness (+55pp raw vs +2pp verified), confirming the verifier is doing real work. The remaining Tier-1 gaps reflect structural limits of static analysis.
|
|
202
227
|
|
|
203
|
-
The full calibration corpus (
|
|
228
|
+
The full calibration corpus (232 public AI-native repos across 8 stack categories, 231 in the LLM calibration subset) is documented at the [MACRC Corpus dashboard](https://trinetra.life).
|
|
204
229
|
|
|
205
230
|
---
|
|
206
231
|
|
|
@@ -236,8 +261,15 @@ Default: all dims warn on absent, pass on partial or present.
|
|
|
236
261
|
## Tests
|
|
237
262
|
|
|
238
263
|
```bash
|
|
239
|
-
|
|
240
|
-
|
|
264
|
+
# Unit tests (fast, no network)
|
|
265
|
+
node --test tests/context-verifier.test.mjs tests/remediator.test.mjs tests/syntax-queries.test.mjs
|
|
266
|
+
|
|
267
|
+
# Integration tests (runs full pipeline on demo/bad-express-app)
|
|
268
|
+
node --test tests/integration.test.mjs
|
|
269
|
+
|
|
270
|
+
# All at once
|
|
271
|
+
node --test tests/*.test.mjs
|
|
272
|
+
# 43 tests, 0 failures
|
|
241
273
|
```
|
|
242
274
|
|
|
243
275
|
---
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Contextual verification for HIGH-FP grep patterns.
|
|
3
3
|
*
|
|
4
|
-
*
|
|
5
|
-
*
|
|
6
|
-
*
|
|
4
|
+
* For each regex match, reads ±N surrounding lines to determine whether the
|
|
5
|
+
* hit is at a real call site rather than a schema definition, test fixture,
|
|
6
|
+
* comment, or README. Rule-based string checks on the context window.
|
|
7
7
|
*
|
|
8
|
-
*
|
|
8
|
+
* Not an AST parser: no tree-sitter or Babel. Fast and dependency-free.
|
|
9
9
|
*
|
|
10
10
|
* Exports:
|
|
11
11
|
* HIGH_FP_KEYS set of pattern keys that need verification
|
package/cli/lib/heuristic.mjs
CHANGED
|
@@ -10,7 +10,8 @@
|
|
|
10
10
|
import { readFileSync, existsSync } from 'fs';
|
|
11
11
|
import { resolve } from 'path';
|
|
12
12
|
import { execSync } from 'child_process';
|
|
13
|
-
import { grepSourcesVerified } from './
|
|
13
|
+
import { grepSourcesVerified } from './context-verifier.mjs';
|
|
14
|
+
import { queryTimeoutHandling, queryDistributedAssumptions } from './syntax-queries.mjs';
|
|
14
15
|
|
|
15
16
|
export const PENDING_DIM = { state: 'pending', severity: null, notes: '', snippets: [] };
|
|
16
17
|
export const ABSENT_DIM = { state: 'absent', severity: null, notes: '', snippets: [] };
|
|
@@ -383,6 +384,8 @@ export function computeHeuristicSummary(layers) {
|
|
|
383
384
|
* stackCategory: one of ai-web-app, api-proxy, mcp-server, agent-framework,
|
|
384
385
|
* cli-tool, desktop-native, plugin-extension, skills-config, or ''
|
|
385
386
|
*/
|
|
387
|
+
const JS_TS_RE = /\.(js|ts|mjs|cjs|jsx|tsx)$/;
|
|
388
|
+
|
|
386
389
|
export function runHeuristic(cloneDir, stackCategory = '', { verified = true } = {}) {
|
|
387
390
|
const deps = parseDeps(cloneDir);
|
|
388
391
|
// verified=true uses context-window analysis for HIGH-FP patterns,
|
|
@@ -391,5 +394,47 @@ export function runHeuristic(cloneDir, stackCategory = '', { verified = true } =
|
|
|
391
394
|
? grepSourcesVerified(cloneDir, GREP_PATTERNS)
|
|
392
395
|
: grepSources(cloneDir);
|
|
393
396
|
const configs = scanConfigs(cloneDir);
|
|
394
|
-
|
|
397
|
+
const layers = heuristicScore(deps, hits, configs, stackCategory);
|
|
398
|
+
|
|
399
|
+
// AST refinement: run acorn syntax queries on JS/TS files for the two
|
|
400
|
+
// dimensions with the largest verified-heuristic gap (+52pp / +20pp).
|
|
401
|
+
// Only runs when verified=true; replaces the grep result when non-null.
|
|
402
|
+
if (verified) {
|
|
403
|
+
let sourceFiles = [];
|
|
404
|
+
try {
|
|
405
|
+
sourceFiles = execSync('git ls-files', { cwd: cloneDir, timeout: 10000 })
|
|
406
|
+
.toString().split('\n')
|
|
407
|
+
.filter(f => f && JS_TS_RE.test(f))
|
|
408
|
+
.map(f => resolve(cloneDir, f));
|
|
409
|
+
} catch {}
|
|
410
|
+
|
|
411
|
+
// timeout_handling — merge per-file AST results
|
|
412
|
+
const timeoutResults = sourceFiles.map(f => queryTimeoutHandling(f)).filter(Boolean);
|
|
413
|
+
if (timeoutResults.length > 0) {
|
|
414
|
+
const totalCalls = timeoutResults.reduce((s, r) => s + r.callsFound, 0);
|
|
415
|
+
const timedOutCalls = timeoutResults.reduce((s, r) => s + r.callsWithTimeout, 0);
|
|
416
|
+
if (timedOutCalls === 0) {
|
|
417
|
+
layers.reliability.timeout_handling = { state: 'absent', severity: 'high', notes: `AST: ${totalCalls} HTTP call(s) with no timeout config`, snippets: [] };
|
|
418
|
+
} else if (timedOutCalls < totalCalls) {
|
|
419
|
+
layers.reliability.timeout_handling = { state: 'partial', severity: 'high', notes: `AST: ${timedOutCalls}/${totalCalls} HTTP calls have timeout config`, snippets: [] };
|
|
420
|
+
} else {
|
|
421
|
+
layers.reliability.timeout_handling = { state: 'present', severity: null, notes: `AST: all ${totalCalls} HTTP call(s) have timeout config`, snippets: [] };
|
|
422
|
+
}
|
|
423
|
+
}
|
|
424
|
+
|
|
425
|
+
// distributed_assumptions — absent if any file has hardcoded localhost in a call arg
|
|
426
|
+
if (!isNA(stackCategory, 'distributed_assumptions')) {
|
|
427
|
+
const distResults = sourceFiles.map(f => queryDistributedAssumptions(f)).filter(Boolean);
|
|
428
|
+
if (distResults.length > 0) {
|
|
429
|
+
const urls = distResults.flatMap(r => r.hardcodedUrls);
|
|
430
|
+
layers.scale_contract.distributed_assumptions = {
|
|
431
|
+
state: 'absent', severity: 'high',
|
|
432
|
+
notes: `AST: hardcoded localhost URL(s) in call arguments: ${[...new Set(urls)].slice(0, 3).join(', ')}`,
|
|
433
|
+
snippets: [],
|
|
434
|
+
};
|
|
435
|
+
}
|
|
436
|
+
}
|
|
437
|
+
}
|
|
438
|
+
|
|
439
|
+
return layers;
|
|
395
440
|
}
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Syntax-tree queries for Tier-1 dimensions with high grep false-positive rates.
|
|
3
|
+
*
|
|
4
|
+
* Uses acorn (pure-JS ES2022 parser, no native deps) for JS/TS files.
|
|
5
|
+
* Returns null when the file is not parseable or has no relevant calls —
|
|
6
|
+
* callers should fall back to the grep result in that case.
|
|
7
|
+
*
|
|
8
|
+
* Exported functions:
|
|
9
|
+
* queryTimeoutHandling(filePath) → { state, callsFound, callsWithTimeout } | null
|
|
10
|
+
* queryDistributedAssumptions(filePath) → { state, hardcodedUrls } | null
|
|
11
|
+
*/
|
|
12
|
+
|
|
13
|
+
import { readFileSync } from 'fs';
|
|
14
|
+
import { createRequire } from 'module';
|
|
15
|
+
|
|
16
|
+
// acorn is a CJS package — load via createRequire
|
|
17
|
+
const require = createRequire(import.meta.url);
|
|
18
|
+
const { parse } = require('acorn');
|
|
19
|
+
|
|
20
|
+
const PARSE_OPTS = { ecmaVersion: 2022, sourceType: 'module' };
|
|
21
|
+
|
|
22
|
+
// ── AST walker — depth-first ───────────────────────────────────────────────────
|
|
23
|
+
|
|
24
|
+
function walk(node, visit, parent = null) {
|
|
25
|
+
if (!node || typeof node !== 'object' || !node.type) return;
|
|
26
|
+
visit(node, parent);
|
|
27
|
+
for (const key of Object.keys(node)) {
|
|
28
|
+
const child = node[key];
|
|
29
|
+
if (Array.isArray(child)) {
|
|
30
|
+
child.forEach(c => walk(c, visit, node));
|
|
31
|
+
} else if (child && typeof child === 'object' && child.type) {
|
|
32
|
+
walk(child, visit, node);
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
// Return callee name as a string, e.g. "axios.get" or "fetch"
|
|
38
|
+
function getCalleeName(callee) {
|
|
39
|
+
if (!callee) return '';
|
|
40
|
+
if (callee.type === 'Identifier') return callee.name;
|
|
41
|
+
if (callee.type === 'MemberExpression') {
|
|
42
|
+
const obj = getCalleeName(callee.object);
|
|
43
|
+
const prop = callee.computed ? '' : (callee.property?.name ?? '');
|
|
44
|
+
return prop ? `${obj}.${prop}` : obj;
|
|
45
|
+
}
|
|
46
|
+
return '';
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
// Known HTTP client call patterns that accept a config object with timeout/signal
|
|
50
|
+
const HTTP_CALLEE_RE = /^(axios\.(get|post|put|delete|patch|request)|fetch)$/;
|
|
51
|
+
|
|
52
|
+
// Keys that signal timeout configuration in an options object
|
|
53
|
+
const TIMEOUT_KEYS = new Set(['timeout', 'signal']);
|
|
54
|
+
|
|
55
|
+
function hasTimeoutOption(args) {
|
|
56
|
+
return args.some(arg =>
|
|
57
|
+
arg.type === 'ObjectExpression' &&
|
|
58
|
+
arg.properties.some(p => {
|
|
59
|
+
const key = p.key?.name ?? p.key?.value ?? '';
|
|
60
|
+
return TIMEOUT_KEYS.has(key);
|
|
61
|
+
})
|
|
62
|
+
);
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
// ── queryTimeoutHandling ───────────────────────────────────────────────────────
|
|
66
|
+
|
|
67
|
+
/**
|
|
68
|
+
* Parse a JS/TS file and check whether axios/fetch calls have timeout config.
|
|
69
|
+
*
|
|
70
|
+
* Returns:
|
|
71
|
+
* null – file not parseable or no HTTP calls found
|
|
72
|
+
* { state: 'absent', callsFound, callsWithTimeout }
|
|
73
|
+
* { state: 'partial', callsFound, callsWithTimeout }
|
|
74
|
+
* { state: 'present', callsFound, callsWithTimeout }
|
|
75
|
+
*/
|
|
76
|
+
export function queryTimeoutHandling(filePath) {
|
|
77
|
+
let src;
|
|
78
|
+
try { src = readFileSync(filePath, 'utf8'); } catch { return null; }
|
|
79
|
+
|
|
80
|
+
let ast;
|
|
81
|
+
try { ast = parse(src, PARSE_OPTS); } catch {
|
|
82
|
+
// Try script mode as fallback (CJS files)
|
|
83
|
+
try { ast = parse(src, { ...PARSE_OPTS, sourceType: 'script' }); } catch { return null; }
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
let callsFound = 0;
|
|
87
|
+
let callsWithTimeout = 0;
|
|
88
|
+
|
|
89
|
+
walk(ast, node => {
|
|
90
|
+
if (node.type !== 'CallExpression') return;
|
|
91
|
+
const name = getCalleeName(node.callee);
|
|
92
|
+
if (!HTTP_CALLEE_RE.test(name)) return;
|
|
93
|
+
callsFound++;
|
|
94
|
+
if (hasTimeoutOption(node.arguments)) callsWithTimeout++;
|
|
95
|
+
});
|
|
96
|
+
|
|
97
|
+
if (callsFound === 0) return null;
|
|
98
|
+
if (callsWithTimeout === 0) return { state: 'absent', callsFound, callsWithTimeout };
|
|
99
|
+
if (callsWithTimeout < callsFound) return { state: 'partial', callsFound, callsWithTimeout };
|
|
100
|
+
return { state: 'present', callsFound, callsWithTimeout };
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
// ── queryDistributedAssumptions ────────────────────────────────────────────────
|
|
104
|
+
|
|
105
|
+
const LOCALHOST_RE = /^https?:\/\/(localhost|127\.0\.0\.1)(:\d+)?/;
|
|
106
|
+
|
|
107
|
+
// Test-block callee names that make a localhost string safe to ignore
|
|
108
|
+
const TEST_CALLEE_NAMES = new Set(['describe', 'it', 'test', 'beforeEach', 'afterEach', 'before', 'after', 'beforeAll', 'afterAll']);
|
|
109
|
+
|
|
110
|
+
/**
|
|
111
|
+
* Parse a JS/TS file and check whether localhost string literals appear
|
|
112
|
+
* as call arguments outside of test blocks.
|
|
113
|
+
*
|
|
114
|
+
* Returns:
|
|
115
|
+
* null – not parseable or no localhost in call args
|
|
116
|
+
* { state: 'absent', hardcodedUrls }
|
|
117
|
+
*/
|
|
118
|
+
export function queryDistributedAssumptions(filePath) {
|
|
119
|
+
let src;
|
|
120
|
+
try { src = readFileSync(filePath, 'utf8'); } catch { return null; }
|
|
121
|
+
|
|
122
|
+
let ast;
|
|
123
|
+
try { ast = parse(src, PARSE_OPTS); } catch {
|
|
124
|
+
try { ast = parse(src, { ...PARSE_OPTS, sourceType: 'script' }); } catch { return null; }
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
// Build parent map so we can walk ancestors
|
|
128
|
+
const parentMap = new Map();
|
|
129
|
+
walk(ast, (node, parent) => { if (parent) parentMap.set(node, parent); });
|
|
130
|
+
|
|
131
|
+
function isInsideTestBlock(node) {
|
|
132
|
+
let cur = parentMap.get(node);
|
|
133
|
+
while (cur) {
|
|
134
|
+
if (cur.type === 'CallExpression') {
|
|
135
|
+
const name = getCalleeName(cur.callee);
|
|
136
|
+
if (TEST_CALLEE_NAMES.has(name)) return true;
|
|
137
|
+
}
|
|
138
|
+
cur = parentMap.get(cur);
|
|
139
|
+
}
|
|
140
|
+
return false;
|
|
141
|
+
}
|
|
142
|
+
|
|
143
|
+
function isCallArgument(node) {
|
|
144
|
+
const parent = parentMap.get(node);
|
|
145
|
+
if (!parent) return false;
|
|
146
|
+
if (parent.type === 'CallExpression') {
|
|
147
|
+
return parent.arguments.includes(node);
|
|
148
|
+
}
|
|
149
|
+
// Template literal expression inside a call
|
|
150
|
+
if (parent.type === 'TemplateLiteral') {
|
|
151
|
+
const grandparent = parentMap.get(parent);
|
|
152
|
+
if (grandparent?.type === 'CallExpression' && grandparent.arguments.includes(parent)) return true;
|
|
153
|
+
}
|
|
154
|
+
return false;
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
const hardcodedUrls = [];
|
|
158
|
+
|
|
159
|
+
walk(ast, node => {
|
|
160
|
+
if (node.type !== 'Literal' || typeof node.value !== 'string') return;
|
|
161
|
+
if (!LOCALHOST_RE.test(node.value)) return;
|
|
162
|
+
if (isInsideTestBlock(node)) return;
|
|
163
|
+
if (!isCallArgument(node)) return;
|
|
164
|
+
hardcodedUrls.push(node.value);
|
|
165
|
+
});
|
|
166
|
+
|
|
167
|
+
if (hardcodedUrls.length === 0) return null;
|
|
168
|
+
return { state: 'absent', hardcodedUrls };
|
|
169
|
+
}
|
package/package.json
CHANGED
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@shubh000/macrc",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "1.0.0",
|
|
4
4
|
"description": "Structural audit CLI for AI-generated code: 20 dimensions across Security, Reliability, Observability, and Scale Contract",
|
|
5
5
|
"type": "module",
|
|
6
|
-
"bin":
|
|
6
|
+
"bin": {
|
|
7
|
+
"macrc": "cli/macrc.mjs"
|
|
8
|
+
},
|
|
7
9
|
"files": [
|
|
8
10
|
"cli/",
|
|
9
11
|
"demo/",
|
|
@@ -16,6 +18,9 @@
|
|
|
16
18
|
"scripts": {
|
|
17
19
|
"test": "node --test tests/*.test.mjs"
|
|
18
20
|
},
|
|
21
|
+
"dependencies": {
|
|
22
|
+
"acorn": "^8.11.0"
|
|
23
|
+
},
|
|
19
24
|
"keywords": [
|
|
20
25
|
"audit",
|
|
21
26
|
"ai",
|
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* Unit tests for cli/lib/
|
|
3
|
-
* Run: node --test tests/
|
|
2
|
+
* Unit tests for cli/lib/context-verifier.mjs
|
|
3
|
+
* Run: node --test tests/context-verifier.test.mjs
|
|
4
4
|
*/
|
|
5
5
|
import { test } from 'node:test';
|
|
6
6
|
import assert from 'node:assert/strict';
|
|
7
|
-
import { verifyMatch } from '../cli/lib/
|
|
7
|
+
import { verifyMatch } from '../cli/lib/context-verifier.mjs';
|
|
8
8
|
|
|
9
9
|
// ── zod_usage ─────────────────────────────────────────────────────────────────
|
|
10
10
|
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
{
|
|
2
|
+
"schema_version": "1.0",
|
|
3
|
+
"repo_id": "fixture",
|
|
4
|
+
"audited_at": "2026-01-02",
|
|
5
|
+
"auditor": "macrc-heuristic",
|
|
6
|
+
"layers": {
|
|
7
|
+
"security": {
|
|
8
|
+
"auth": { "state": "present", "severity": null, "notes": "fixture: auth wired", "snippets": [] },
|
|
9
|
+
"input_validation": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
10
|
+
"transport_security": { "state": "absent", "severity": null, "notes": "fixture", "snippets": [] },
|
|
11
|
+
"dependency_hygiene": { "state": "pending", "severity": null, "notes": "fixture", "snippets": [] },
|
|
12
|
+
"secrets_management": { "state": "pending", "severity": null, "notes": "fixture", "snippets": [] }
|
|
13
|
+
},
|
|
14
|
+
"reliability": {
|
|
15
|
+
"error_handling": { "state": "absent", "severity": "critical", "notes": "fixture", "snippets": [] },
|
|
16
|
+
"retry_logic": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
17
|
+
"circuit_breaking": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
18
|
+
"graceful_degradation": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] },
|
|
19
|
+
"timeout_handling": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] }
|
|
20
|
+
},
|
|
21
|
+
"observability": {
|
|
22
|
+
"logging": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
23
|
+
"metrics_emission": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] },
|
|
24
|
+
"tracing": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] },
|
|
25
|
+
"alerting": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] },
|
|
26
|
+
"health_checks": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] }
|
|
27
|
+
},
|
|
28
|
+
"scale_contract": {
|
|
29
|
+
"statelessness": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
30
|
+
"load_balancing_awareness": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] },
|
|
31
|
+
"distributed_assumptions": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
32
|
+
"rate_limiting": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
33
|
+
"caching_strategy": { "state": "absent", "severity": "low", "notes": "fixture", "snippets": [] }
|
|
34
|
+
}
|
|
35
|
+
}
|
|
36
|
+
}
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
{
|
|
2
|
+
"schema_version": "1.0",
|
|
3
|
+
"repo_id": "fixture",
|
|
4
|
+
"audited_at": "2026-01-01",
|
|
5
|
+
"auditor": "macrc-heuristic",
|
|
6
|
+
"layers": {
|
|
7
|
+
"security": {
|
|
8
|
+
"auth": { "state": "absent", "severity": "high", "notes": "fixture: no auth", "snippets": [] },
|
|
9
|
+
"input_validation": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
10
|
+
"transport_security": { "state": "absent", "severity": null, "notes": "fixture", "snippets": [] },
|
|
11
|
+
"dependency_hygiene": { "state": "pending", "severity": null, "notes": "fixture", "snippets": [] },
|
|
12
|
+
"secrets_management": { "state": "pending", "severity": null, "notes": "fixture", "snippets": [] }
|
|
13
|
+
},
|
|
14
|
+
"reliability": {
|
|
15
|
+
"error_handling": { "state": "absent", "severity": "critical", "notes": "fixture", "snippets": [] },
|
|
16
|
+
"retry_logic": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
17
|
+
"circuit_breaking": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
18
|
+
"graceful_degradation": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] },
|
|
19
|
+
"timeout_handling": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] }
|
|
20
|
+
},
|
|
21
|
+
"observability": {
|
|
22
|
+
"logging": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
23
|
+
"metrics_emission": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] },
|
|
24
|
+
"tracing": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] },
|
|
25
|
+
"alerting": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] },
|
|
26
|
+
"health_checks": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] }
|
|
27
|
+
},
|
|
28
|
+
"scale_contract": {
|
|
29
|
+
"statelessness": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
30
|
+
"load_balancing_awareness": { "state": "absent", "severity": "medium", "notes": "fixture", "snippets": [] },
|
|
31
|
+
"distributed_assumptions": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
32
|
+
"rate_limiting": { "state": "absent", "severity": "high", "notes": "fixture", "snippets": [] },
|
|
33
|
+
"caching_strategy": { "state": "absent", "severity": "low", "notes": "fixture", "snippets": [] }
|
|
34
|
+
}
|
|
35
|
+
}
|
|
36
|
+
}
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
import { test } from 'node:test';
|
|
2
|
+
import assert from 'node:assert/strict';
|
|
3
|
+
import { resolve } from 'path';
|
|
4
|
+
import { spawnSync } from 'child_process';
|
|
5
|
+
import { fileURLToPath } from 'url';
|
|
6
|
+
import { runHeuristic } from '../cli/lib/heuristic.mjs';
|
|
7
|
+
import { buildPatches } from '../cli/lib/remediator.mjs';
|
|
8
|
+
|
|
9
|
+
const __dir = fileURLToPath(new URL('.', import.meta.url));
|
|
10
|
+
const DEMO = resolve(__dir, '../demo/bad-express-app');
|
|
11
|
+
|
|
12
|
+
// ── Absent-dim assertions ──────────────────────────────────────────────────────
|
|
13
|
+
// bad-express-app:
|
|
14
|
+
// no auth packages/patterns → secEnhancement.auth absent
|
|
15
|
+
// no validation packages/patterns → secEnhancement.input_validation absent
|
|
16
|
+
// no retry packages/patterns → reliability.retry_logic absent
|
|
17
|
+
// no timeout patterns → reliability.timeout_handling absent
|
|
18
|
+
// no health route → observability.health_checks absent
|
|
19
|
+
// express-session, no redis → scale_contract.statelessness absent
|
|
20
|
+
// localhost URLs in api.js → scale_contract.distributed_assumptions absent
|
|
21
|
+
// no rate-limit packages → scale_contract.rate_limiting absent
|
|
22
|
+
|
|
23
|
+
const ABSENT = [
|
|
24
|
+
['secEnhancement', 'auth'],
|
|
25
|
+
['secEnhancement', 'input_validation'],
|
|
26
|
+
['reliability', 'retry_logic'],
|
|
27
|
+
['reliability', 'timeout_handling'],
|
|
28
|
+
['observability', 'health_checks'],
|
|
29
|
+
['scale_contract', 'statelessness'],
|
|
30
|
+
['scale_contract', 'distributed_assumptions'],
|
|
31
|
+
['scale_contract', 'rate_limiting'],
|
|
32
|
+
];
|
|
33
|
+
|
|
34
|
+
// Run once; reuse across all tests
|
|
35
|
+
const heuristicResult = runHeuristic(DEMO, 'ai-web-app', { verified: true });
|
|
36
|
+
|
|
37
|
+
for (const [layer, dim] of ABSENT) {
|
|
38
|
+
test(`${layer}.${dim} is absent in bad-express-app`, () => {
|
|
39
|
+
const dimResult = heuristicResult[layer]?.[dim];
|
|
40
|
+
// secEnhancement only records present/partial — undefined means absent
|
|
41
|
+
const state = dimResult?.state ?? 'absent';
|
|
42
|
+
assert.equal(state, 'absent', `expected absent, got "${state}"`);
|
|
43
|
+
});
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
// ── Patch count test ───────────────────────────────────────────────────────────
|
|
47
|
+
// buildPatches expects auditResult.layers with 'security' (not 'secEnhancement')
|
|
48
|
+
|
|
49
|
+
test('buildPatches produces ≥7 patches on bad-express-app audit', () => {
|
|
50
|
+
const security = {
|
|
51
|
+
auth: heuristicResult.secEnhancement.auth || { state: 'absent', severity: null, notes: '', snippets: [] },
|
|
52
|
+
input_validation: heuristicResult.secEnhancement.input_validation || { state: 'absent', severity: null, notes: '', snippets: [] },
|
|
53
|
+
transport_security: heuristicResult.secEnhancement.transport_security || { state: 'absent', severity: null, notes: '', snippets: [] },
|
|
54
|
+
dependency_hygiene: { state: 'pending', severity: null, notes: '', snippets: [] },
|
|
55
|
+
secrets_management: { state: 'pending', severity: null, notes: '', snippets: [] },
|
|
56
|
+
};
|
|
57
|
+
const auditResult = {
|
|
58
|
+
layers: {
|
|
59
|
+
security,
|
|
60
|
+
reliability: heuristicResult.reliability,
|
|
61
|
+
observability: heuristicResult.observability,
|
|
62
|
+
scale_contract: heuristicResult.scale_contract,
|
|
63
|
+
},
|
|
64
|
+
};
|
|
65
|
+
const patches = buildPatches(auditResult, DEMO);
|
|
66
|
+
assert.ok(patches.length >= 7, `expected ≥7 patches, got ${patches.length}: ${patches.map(p => p._dim).join(', ')}`);
|
|
67
|
+
});
|
|
68
|
+
|
|
69
|
+
// ── Diff regression test ───────────────────────────────────────────────────────
|
|
70
|
+
|
|
71
|
+
test('diff detects auth absent→present as IMPROVED', () => {
|
|
72
|
+
const FIX_BEFORE = resolve(__dir, 'fixtures/diff-before.json');
|
|
73
|
+
const FIX_AFTER = resolve(__dir, 'fixtures/diff-after.json');
|
|
74
|
+
const MACRC = resolve(__dir, '../cli/macrc.mjs');
|
|
75
|
+
|
|
76
|
+
const r = spawnSync(process.execPath, [MACRC, 'diff', FIX_BEFORE, FIX_AFTER], { encoding: 'utf8' });
|
|
77
|
+
assert.equal(r.status, 0, `CLI exited ${r.status}: ${r.stderr}`);
|
|
78
|
+
assert.ok(
|
|
79
|
+
r.stdout.includes('IMPROVED') && r.stdout.includes('auth'),
|
|
80
|
+
`expected "IMPROVED" and "auth" in diff output, got:\n${r.stdout}`
|
|
81
|
+
);
|
|
82
|
+
});
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
import { test } from 'node:test';
|
|
2
|
+
import assert from 'node:assert/strict';
|
|
3
|
+
import { writeFileSync, unlinkSync } from 'fs';
|
|
4
|
+
import { resolve } from 'path';
|
|
5
|
+
import { fileURLToPath } from 'url';
|
|
6
|
+
import { queryTimeoutHandling, queryDistributedAssumptions } from '../cli/lib/syntax-queries.mjs';
|
|
7
|
+
|
|
8
|
+
const TMP_DIR = fileURLToPath(new URL('.', import.meta.url));
|
|
9
|
+
|
|
10
|
+
function tmp(name, src) {
|
|
11
|
+
const p = resolve(TMP_DIR, `_tmp_sq_${name}.js`);
|
|
12
|
+
writeFileSync(p, src);
|
|
13
|
+
return p;
|
|
14
|
+
}
|
|
15
|
+
function rm(p) { try { unlinkSync(p); } catch {} }
|
|
16
|
+
|
|
17
|
+
// ── queryTimeoutHandling ───────────────────────────────────────────────────────
|
|
18
|
+
|
|
19
|
+
test('timeout_handling: absent when axios.get has no timeout', () => {
|
|
20
|
+
const p = tmp('t1', `import axios from 'axios';\naxios.get('http://api.example.com/data');`);
|
|
21
|
+
const r = queryTimeoutHandling(p);
|
|
22
|
+
rm(p);
|
|
23
|
+
assert.equal(r?.state, 'absent');
|
|
24
|
+
assert.equal(r?.callsFound, 1);
|
|
25
|
+
assert.equal(r?.callsWithTimeout, 0);
|
|
26
|
+
});
|
|
27
|
+
|
|
28
|
+
test('timeout_handling: present when axios.get has { timeout }', () => {
|
|
29
|
+
const p = tmp('t2', `import axios from 'axios';\naxios.get('http://api.example.com/data', { timeout: 5000 });`);
|
|
30
|
+
const r = queryTimeoutHandling(p);
|
|
31
|
+
rm(p);
|
|
32
|
+
assert.equal(r?.state, 'present');
|
|
33
|
+
assert.equal(r?.callsWithTimeout, 1);
|
|
34
|
+
});
|
|
35
|
+
|
|
36
|
+
test('timeout_handling: present when fetch has AbortSignal in { signal }', () => {
|
|
37
|
+
const p = tmp('t3', `fetch('http://api.example.com', { signal: AbortSignal.timeout(5000) });`);
|
|
38
|
+
const r = queryTimeoutHandling(p);
|
|
39
|
+
rm(p);
|
|
40
|
+
assert.equal(r?.state, 'present');
|
|
41
|
+
});
|
|
42
|
+
|
|
43
|
+
test('timeout_handling: null for file with no HTTP calls', () => {
|
|
44
|
+
const p = tmp('t4', `const x = 1 + 2;\nconsole.log(x);`);
|
|
45
|
+
const r = queryTimeoutHandling(p);
|
|
46
|
+
rm(p);
|
|
47
|
+
assert.equal(r, null);
|
|
48
|
+
});
|
|
49
|
+
|
|
50
|
+
test('timeout_handling: partial when mixed (some with, some without)', () => {
|
|
51
|
+
const p = tmp('t5', [
|
|
52
|
+
`import axios from 'axios';`,
|
|
53
|
+
`axios.get('/a');`,
|
|
54
|
+
`axios.get('/b', { timeout: 3000 });`,
|
|
55
|
+
].join('\n'));
|
|
56
|
+
const r = queryTimeoutHandling(p);
|
|
57
|
+
rm(p);
|
|
58
|
+
assert.equal(r?.state, 'partial');
|
|
59
|
+
assert.equal(r?.callsFound, 2);
|
|
60
|
+
assert.equal(r?.callsWithTimeout, 1);
|
|
61
|
+
});
|
|
62
|
+
|
|
63
|
+
// ── queryDistributedAssumptions ────────────────────────────────────────────────
|
|
64
|
+
|
|
65
|
+
test('distributed_assumptions: absent when localhost in call argument', () => {
|
|
66
|
+
const p = tmp('d1', `fetch('http://localhost:3000/api');`);
|
|
67
|
+
const r = queryDistributedAssumptions(p);
|
|
68
|
+
rm(p);
|
|
69
|
+
assert.equal(r?.state, 'absent');
|
|
70
|
+
assert.ok(r?.hardcodedUrls.length >= 1);
|
|
71
|
+
});
|
|
72
|
+
|
|
73
|
+
test('distributed_assumptions: null when localhost not in call argument', () => {
|
|
74
|
+
const p = tmp('d2', `const URL = 'http://localhost:3000';`);
|
|
75
|
+
const r = queryDistributedAssumptions(p);
|
|
76
|
+
rm(p);
|
|
77
|
+
assert.equal(r, null);
|
|
78
|
+
});
|
|
79
|
+
|
|
80
|
+
test('distributed_assumptions: null when localhost in test block', () => {
|
|
81
|
+
const p = tmp('d3', [
|
|
82
|
+
`describe('test', () => {`,
|
|
83
|
+
` it('calls local', () => {`,
|
|
84
|
+
` fetch('http://localhost:3000/api');`,
|
|
85
|
+
` });`,
|
|
86
|
+
`});`,
|
|
87
|
+
].join('\n'));
|
|
88
|
+
const r = queryDistributedAssumptions(p);
|
|
89
|
+
rm(p);
|
|
90
|
+
assert.equal(r, null);
|
|
91
|
+
});
|
|
92
|
+
|
|
93
|
+
test('distributed_assumptions: absent for 127.0.0.1 in call argument', () => {
|
|
94
|
+
const p = tmp('d4', `axios.get('http://127.0.0.1:8080/health');`);
|
|
95
|
+
const r = queryDistributedAssumptions(p);
|
|
96
|
+
rm(p);
|
|
97
|
+
assert.equal(r?.state, 'absent');
|
|
98
|
+
});
|