crewforth 0.0.1 → 3.0.0-rc.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +3805 -0
- package/LICENSE +21 -0
- package/README.md +37 -7
- package/VERSION +1 -0
- package/adopt.sh +1739 -0
- package/bin/cli.js +497 -0
- package/kit/AGENT_TEMPLATE.md +92 -0
- package/kit/CLAUDE.md +228 -0
- package/kit/README.md +107 -0
- package/kit/agents/crew-backend-expert.md +95 -0
- package/kit/agents/crew-commit-agent.md +96 -0
- package/kit/agents/crew-database-expert.md +84 -0
- package/kit/agents/crew-devops-expert.md +82 -0
- package/kit/agents/crew-frontend-expert.md +85 -0
- package/kit/agents/crew-performance-expert.md +78 -0
- package/kit/agents/crew-planner.md +60 -0
- package/kit/agents/crew-privacy-agent.md +71 -0
- package/kit/agents/crew-review-agent.md +95 -0
- package/kit/agents/crew-security-expert.md +73 -0
- package/kit/agents/crew-session-manager.md +77 -0
- package/kit/agents/crew-test-expert.md +64 -0
- package/kit/eval/doctor.sh +639 -0
- package/kit/eval/gate-report.sh +137 -0
- package/kit/eval/golden-holdout.txt +53 -0
- package/kit/eval/golden-routing.txt +199 -0
- package/kit/eval/lib/crew-env.sh +14 -0
- package/kit/eval/lib/settings-json.awk +154 -0
- package/kit/eval/lib/skill-listing.awk +43 -0
- package/kit/eval/lib/star.sh +70 -0
- package/kit/eval/parser-conformance.sh +612 -0
- package/kit/eval/preflight.sh +186 -0
- package/kit/eval/routing-eval.sh +209 -0
- package/kit/eval/scan-skill.sh +162 -0
- package/kit/eval/smoke-test.sh +6736 -0
- package/kit/eval/update-guard.sh +81 -0
- package/kit/eval/utilization.sh +130 -0
- package/kit/hooks/board-sync.sh +106 -0
- package/kit/hooks/board.sh +962 -0
- package/kit/hooks/commit-msg +49 -0
- package/kit/hooks/context-usage.sh +353 -0
- package/kit/hooks/floor-blocklist.txt +124 -0
- package/kit/hooks/guard-bash.sh +1433 -0
- package/kit/hooks/guard-commit-scan.sh +511 -0
- package/kit/hooks/guard-write.sh +513 -0
- package/kit/hooks/pre-commit +577 -0
- package/kit/hooks/route-hint.sh +185 -0
- package/kit/hooks/secret-blocklist.txt +82 -0
- package/kit/hooks/session-guard.sh +149 -0
- package/kit/hooks/session-rehydrate.sh +41 -0
- package/kit/hooks/session-stats.sh +252 -0
- package/kit/hooks/session-update-check.sh +236 -0
- package/kit/hooks/skill-trust.sh +163 -0
- package/kit/hooks/trace-blocklist.txt +53 -0
- package/kit/settings.json +132 -0
- package/kit/skills/a11y/SKILL.md +44 -0
- package/kit/skills/adr/SKILL.md +74 -0
- package/kit/skills/api-design/SKILL.md +50 -0
- package/kit/skills/automode-policy/SKILL.md +91 -0
- package/kit/skills/automode-policy/references/policy.json +13 -0
- package/kit/skills/automode-policy/scripts/apply.sh +84 -0
- package/kit/skills/automode-policy/scripts/check.sh +102 -0
- package/kit/skills/backend-architecture/SKILL.md +103 -0
- package/kit/skills/brainstorm/SKILL.md +47 -0
- package/kit/skills/ci-pipeline/SKILL.md +38 -0
- package/kit/skills/commit-message/SKILL.md +82 -0
- package/kit/skills/commit-message/references/examples.md +8 -0
- package/kit/skills/confidence-check/SKILL.md +86 -0
- package/kit/skills/crew-board/SKILL.md +55 -0
- package/kit/skills/crew-brainstorm/SKILL.md +19 -0
- package/kit/skills/crew-code-review/SKILL.md +151 -0
- package/kit/skills/crew-code-review/references/history-search.md +19 -0
- package/kit/skills/crew-code-review/references/panel-mode.md +32 -0
- package/kit/skills/crew-code-review/references/receiving.md +23 -0
- package/kit/skills/crew-doctor/SKILL.md +46 -0
- package/kit/skills/crew-gates/SKILL.md +25 -0
- package/kit/skills/crew-handoff/SKILL.md +14 -0
- package/kit/skills/crew-plan/SKILL.md +18 -0
- package/kit/skills/crew-review/SKILL.md +26 -0
- package/kit/skills/crew-ship/SKILL.md +19 -0
- package/kit/skills/crew-skill/SKILL.md +62 -0
- package/kit/skills/crew-studio/SKILL.md +129 -0
- package/kit/skills/crew-update/SKILL.md +43 -0
- package/kit/skills/db-migration/SKILL.md +120 -0
- package/kit/skills/db-migration/references/backup-restore.md +33 -0
- package/kit/skills/db-migration/references/raw-sql.md +30 -0
- package/kit/skills/db-migration/references/tool-matrix.md +35 -0
- package/kit/skills/dependency-audit/SKILL.md +61 -0
- package/kit/skills/dependency-upgrade/SKILL.md +82 -0
- package/kit/skills/deploy/SKILL.md +100 -0
- package/kit/skills/deploy/references/proxy-ssl.md +56 -0
- package/kit/skills/deploy/references/rollback.md +18 -0
- package/kit/skills/deploy/references/scripts.md +9 -0
- package/kit/skills/deploy/references/topology-a.md +128 -0
- package/kit/skills/deploy/references/topology-b.md +44 -0
- package/kit/skills/docs-writer/SKILL.md +53 -0
- package/kit/skills/docs-writer/references/coauthoring.md +34 -0
- package/kit/skills/eval-grader/SKILL.md +75 -0
- package/kit/skills/eval-grader/references/method.md +51 -0
- package/kit/skills/frontend/SKILL.md +56 -0
- package/kit/skills/frontend/references/verify-contract.md +41 -0
- package/kit/skills/frontend-design/SKILL.md +71 -0
- package/kit/skills/frontend-design/references/design-review.md +34 -0
- package/kit/skills/frontend-rn-expo/SKILL.md +25 -0
- package/kit/skills/handoff/SKILL.md +71 -0
- package/kit/skills/handoff/references/cold-reader.md +51 -0
- package/kit/skills/i18n-integrity/SKILL.md +31 -0
- package/kit/skills/incident-runbook/SKILL.md +40 -0
- package/kit/skills/incident-runbook/references/postmortem.md +16 -0
- package/kit/skills/iterate/SKILL.md +40 -0
- package/kit/skills/mcp-builder/SKILL.md +54 -0
- package/kit/skills/mcp-builder/references/building.md +71 -0
- package/kit/skills/observability/SKILL.md +50 -0
- package/kit/skills/performance/SKILL.md +38 -0
- package/kit/skills/performance/references/bottlenecks.md +10 -0
- package/kit/skills/privacy-compliance/SKILL.md +75 -0
- package/kit/skills/red-team/SKILL.md +48 -0
- package/kit/skills/reflect/SKILL.md +57 -0
- package/kit/skills/release/SKILL.md +44 -0
- package/kit/skills/security-scan/SKILL.md +99 -0
- package/kit/skills/security-scan/references/ai-agents.md +107 -0
- package/kit/skills/security-scan/references/fronts.md +99 -0
- package/kit/skills/security-scan/references/prompting.md +23 -0
- package/kit/skills/security-scan/references/reporting.md +75 -0
- package/kit/skills/security-scan/references/verify.md +131 -0
- package/kit/skills/sonarqube-check/SKILL.md +108 -0
- package/kit/skills/sonarqube-check/references/local-engines.md +32 -0
- package/kit/skills/sonarqube-check/references/stand-up-server.md +25 -0
- package/kit/skills/spec-planning/SKILL.md +92 -0
- package/kit/skills/systematic-debugging/SKILL.md +57 -0
- package/kit/skills/systematic-debugging/references/techniques.md +44 -0
- package/kit/skills/teamboard/SKILL.md +88 -0
- package/kit/skills/teamboard/references/setup.md +37 -0
- package/kit/skills/testing/SKILL.md +49 -0
- package/kit/skills/testing/references/flaky-triage.md +51 -0
- package/kit/skills/threat-model/SKILL.md +57 -0
- package/kit/skills/threat-model/references/interview.md +51 -0
- package/kit/skills/threat-model/references/schema.md +42 -0
- package/kit/skills/token-budget/SKILL.md +51 -0
- package/kit/skills/trace-scan/SKILL.md +43 -0
- package/kit/skills/worktree/SKILL.md +40 -0
- package/kit/studio/README.md +266 -0
- package/kit/studio/ensure-node.sh +471 -0
- package/kit/studio/package.json +14 -0
- package/kit/studio/server/hooks/studio-gate.sh +169 -0
- package/kit/studio/server/index.js +902 -0
- package/kit/studio/server/lib/crew-env.js +11 -0
- package/kit/studio/server/lib/fleet.js +146 -0
- package/kit/studio/server/lib/graph.js +533 -0
- package/kit/studio/server/lib/instance.js +135 -0
- package/kit/studio/server/lib/kit-telemetry.js +182 -0
- package/kit/studio/server/lib/kit.js +158 -0
- package/kit/studio/server/lib/palette.js +118 -0
- package/kit/studio/server/lib/peers.js +72 -0
- package/kit/studio/server/lib/permissions.js +150 -0
- package/kit/studio/server/lib/projects.js +294 -0
- package/kit/studio/server/lib/roster.js +191 -0
- package/kit/studio/server/lib/session.js +309 -0
- package/kit/studio/server/lib/terminal.js +91 -0
- package/kit/studio/server/lib/transcript.js +100 -0
- package/kit/studio/web/app.js +1098 -0
- package/kit/studio/web/canvas.js +1179 -0
- package/kit/studio/web/chat.js +543 -0
- package/kit/studio/web/index.html +86 -0
- package/kit/studio/web/md.js +110 -0
- package/kit/studio/web/storage-migrate.js +23 -0
- package/kit/studio/web/style.css +1453 -0
- package/package.json +46 -8
- package/start.sh +764 -0
- package/bin/crewforth.js +0 -10
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: reflect
|
|
3
|
+
description: |
|
|
4
|
+
Retrospective self-audit after nontrivial work: unverified assumptions, skipped items, is-this-the-right-
|
|
5
|
+
approach — findings, not code. The step-back counterpart to iterate's refine-to-done loop.
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Reflect — step back and audit the work, not just the code
|
|
9
|
+
|
|
10
|
+
<!-- routing-eval reads the next line; why it sits in the body: AGENT_TEMPLATE.md -->
|
|
11
|
+
Trigger phrases: "reflect", "retro", "retrospective", "what did we miss", "step back", "introspect"
|
|
12
|
+
|
|
13
|
+
## When
|
|
14
|
+
A nontrivial chunk of work just finished (a feature, a plan, a debugging session) and it's worth a deliberate
|
|
15
|
+
step back **before** committing or moving on. This is the meta-cognitive counterpart to [[iterate]]: iterate
|
|
16
|
+
drives a change *to* its exit test; reflect asks whether the exit test — and the approach behind it — was even
|
|
17
|
+
the right one. Skip it for a one-line, unambiguous change; there's nothing to reflect on.
|
|
18
|
+
|
|
19
|
+
## Measure first, then remember
|
|
20
|
+
Run **`bash .claude/hooks/session-stats.sh`** before answering anything below, and open the pass with what it
|
|
21
|
+
reports. A retro built only on recall is an interview with the least reliable witness in the room: the model
|
|
22
|
+
reconstructs a tidy story from a context that has already been summarised, and the stretches where it span on a
|
|
23
|
+
failing approach are exactly the ones it remembers least. The script counts what actually happened — prompts,
|
|
24
|
+
tool calls, failing loops, near-duplicate prompts, interrupts, auto-compactions.
|
|
25
|
+
|
|
26
|
+
Treat each ⚠️ as a **question to answer in the pass**, not a verdict: a runaway loop asks *what assumption kept
|
|
27
|
+
failing*; a repeated prompt asks *what context never landed*; an interrupt asks *where intent diverged*; an auto
|
|
28
|
+
compaction asks *what state was silently dropped*. If the numbers and your recollection disagree, the numbers are
|
|
29
|
+
the record. If the script is missing (a plugin install has no `.claude/hooks/`), say the retro is recall-based —
|
|
30
|
+
do not present recall as measurement.
|
|
31
|
+
|
|
32
|
+
## The pass
|
|
33
|
+
Ask each question honestly and write the answer, not a reassurance:
|
|
34
|
+
1. **Unverified assumptions** — what did I take for granted that I never checked? Name each one and whether it
|
|
35
|
+
was actually confirmed (read the code / ran the flow) or just assumed.
|
|
36
|
+
2. **What got skipped** — an edge case, an error path, a test, a doc update, a security/privacy angle. What was
|
|
37
|
+
silently dropped, and was that a conscious trade-off or an oversight?
|
|
38
|
+
3. **Right approach?** — with hindsight, is this the direction we'd still pick? Did scope creep in? Is there a
|
|
39
|
+
simpler path we walked past ([[code-review]] altitude: could 200 lines be 50)?
|
|
40
|
+
4. **Evidence gap** — which claims of "done" / "works" rest on having *observed* behavior vs. on inference?
|
|
41
|
+
An unobserved "it works" is a finding (see [[iterate]] / verify: drive the real flow).
|
|
42
|
+
5. **What I'd tell the next session** — the one thing a fresh context most needs to know (feeds [[handoff]]).
|
|
43
|
+
|
|
44
|
+
## Guardrails
|
|
45
|
+
- **Findings, not code.** Reflect surfaces gaps; the relevant specialist fixes them (a found gap re-enters
|
|
46
|
+
[[iterate]] or the author agent). It changes nothing itself.
|
|
47
|
+
- **Honest, not performative.** A retro that only confirms good work is a failed retro. If nothing is found,
|
|
48
|
+
say specifically *why* you're confident (what was verified), don't just assert it.
|
|
49
|
+
- **Bounded.** One pass, concrete findings, then act or close — not an open rumination loop.
|
|
50
|
+
- **Token discipline** ([[token-budget]]): a short ranked findings list to the main thread; detail to a file.
|
|
51
|
+
|
|
52
|
+
## DoD (this skill's contribution)
|
|
53
|
+
- `session-stats.sh` was run and every ⚠️ it raised is answered — or its absence is stated.
|
|
54
|
+
- Assumptions are labeled verified vs. assumed; unverified ones are flagged, not buried.
|
|
55
|
+
- Skipped items and any scope creep are named explicitly.
|
|
56
|
+
- Every "done/works" claim is traced to observed evidence or marked as inference.
|
|
57
|
+
- Findings are actionable (each maps to a fix, a follow-up, or an accepted trade-off).
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: release
|
|
3
|
+
description: |
|
|
4
|
+
Versioning and CHANGELOG: SemVer mapped from Conventional Commits, Keep a Changelog format, tagging,
|
|
5
|
+
pre-release gates.
|
|
6
|
+
Use when cutting a version: bumping, writing the CHANGELOG entry, or tagging.
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Release & CHANGELOG
|
|
10
|
+
|
|
11
|
+
<!-- routing-eval reads the next line; why it sits in the body: AGENT_TEMPLATE.md -->
|
|
12
|
+
Trigger phrases: "release", "cut a release", "changelog", "version bump", "bump the version", "new version", "tag", "semver"
|
|
13
|
+
|
|
14
|
+
## SemVer mapping (derive from Conventional Commits)
|
|
15
|
+
- `fix:` → **PATCH** (x.y.Z)
|
|
16
|
+
- `feat:` → **MINOR** (x.Y.0)
|
|
17
|
+
- `BREAKING CHANGE:` / `feat!:` → **MAJOR** (X.0.0)
|
|
18
|
+
|
|
19
|
+
## CHANGELOG (Keep a Changelog)
|
|
20
|
+
Headings: **Added · Changed · Fixed · Removed · Security · Deprecated**.
|
|
21
|
+
Every version is dated; the `Unreleased` section can be auto-populated from commits.
|
|
22
|
+
|
|
23
|
+
## Pre-release gates (all must pass)
|
|
24
|
+
- [ ] Tests green + `sonarqube-check` PASSED
|
|
25
|
+
- [ ] `dependency-audit` clean (0 HIGH/CRITICAL)
|
|
26
|
+
- [ ] CHANGELOG up to date
|
|
27
|
+
- [ ] Version number conforms to SemVer
|
|
28
|
+
- [ ] **The release commit is a diff too (§4.6).** A `chore(release)` bump, a CHANGELOG-only edit and a docs-site
|
|
29
|
+
version touch each stage a diff, and §4.6 exempts nothing by size — `crew-review-agent` runs over them like
|
|
30
|
+
any other. A version that disagrees with the tag, a CHANGELOG entry naming the wrong one, a generated
|
|
31
|
+
manifest edited by hand: that is what this review is for, and all three have shipped before.
|
|
32
|
+
- [ ] **Every distribution channel has a documented way to GET this version.** Publishing and reaching users are
|
|
33
|
+
different events. A channel whose consumers pin at install time — an editor extension, a plugin, a vendored
|
|
34
|
+
copy — leaves them on the version they installed until they ask for a new one, so a security fix ships and
|
|
35
|
+
does not arrive. For each channel name the upgrade command in the README, and say plainly where it is not
|
|
36
|
+
automatic.
|
|
37
|
+
|
|
38
|
+
## Tagging
|
|
39
|
+
```bash
|
|
40
|
+
git tag -a vX.Y.Z -m "vX.Y.Z" # asks for approval (§4.4); push on explicit request (§4.5)
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## DoD
|
|
44
|
+
- Correct SemVer bump; complete CHANGELOG; tag + rollback plan ready.
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: security-scan
|
|
3
|
+
description: |
|
|
4
|
+
Stack-agnostic security audit: map the attack surface, trace untrusted input to dangerous calls, surface
|
|
5
|
+
dependency and configuration flaws. Severity-ranked report with fixes.
|
|
6
|
+
Use when auth, input handling, secrets or dependencies change, and before a release.
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Security Scan
|
|
10
|
+
|
|
11
|
+
<!-- routing-eval reads the next line; why it sits in the body: AGENT_TEMPLATE.md -->
|
|
12
|
+
Trigger phrases: "security scan", "run a security scan", "OWASP check", "scan for vulnerabilities", "find security vulnerabilities", "security audit", "api key", "secret key", "leaked credential", "credentials leaked", "hardcoded secret", "hardcoded password", "sql injection", "can an attacker"
|
|
13
|
+
|
|
14
|
+
The core of a security vulnerability fits in a single sentence: **an untrusted input reaches a
|
|
15
|
+
dangerous operation without being adequately checked.** This skill chases exactly that sentence — it first
|
|
16
|
+
looks for where the input comes from, then where it flows, and what gate should sit in between.
|
|
17
|
+
It is stack-agnostic: whatever the language/framework, the same logic applies; when current tooling and
|
|
18
|
+
patterns are needed, it runs a web search.
|
|
19
|
+
|
|
20
|
+
> **Crewforth adaptation (local, .claude/):** `crew-security-expert` applies this; findings are carried to
|
|
21
|
+
> **crew-review-agent** in severity order, whatever the stack. Automatic
|
|
22
|
+
> fixes only with explicit approval (§4.4); `.claude` does not go to the repo (§4.3). §4 Prohibitions apply.
|
|
23
|
+
|
|
24
|
+
## What it does, what it doesn't
|
|
25
|
+
- **Does:** surfaces common vulnerability classes, known vulnerable dependencies, and risky configuration; ties each finding to a concrete fix.
|
|
26
|
+
- **Doesn't:** does not replace a professional pentest / SAST / DAST. The report **guides**, it does not give full assurance — state this at the end of the report.
|
|
27
|
+
- **Boundary:** analysis is local; code/data is not sent to an external service, and the project directory is not left.
|
|
28
|
+
|
|
29
|
+
## A question is not a scan
|
|
30
|
+
Loading this skill does not mean running all of it. "Is this query injectable?", "should this key be in the
|
|
31
|
+
repo?", "how do I hash these passwords?" are **questions**: answer them from the relevant part of this skill and
|
|
32
|
+
stop — no discovery pass, no fan-out of verifiers, no coverage ledger, no report file. The full workflow below runs
|
|
33
|
+
when the user asks for a scan, an audit or a security review of a codebase or a change, or asks for the report
|
|
34
|
+
itself. When a request could be either, ask one question before starting the full workflow; do not guess upward.
|
|
35
|
+
|
|
36
|
+
## Mental model — source → gate → sink
|
|
37
|
+
Reduce every check to three questions:
|
|
38
|
+
1. **Source** — where does the input enter? (route, API endpoint, form, CLI argument, file upload, WebSocket, queue message, external API response)
|
|
39
|
+
2. **Sink** — which dangerous operation does this input reach? (SQL execution, shell, file path, HTML render, deserialization, template)
|
|
40
|
+
3. **Gate** — is there validation / parameterization / escaping / authorization in between? If not, that's the finding.
|
|
41
|
+
|
|
42
|
+
The scan applies this model on four fronts: **dependency · code · configuration · authorization**. The result is ranked by severity, and the fix is presented for the user to choose.
|
|
43
|
+
|
|
44
|
+
## Scope
|
|
45
|
+
Prefer **`docs/THREAT_MODEL.md`** if present (from the `threat-model` skill): its entry points and attack classes
|
|
46
|
+
are the focus areas, and its impact/likelihood bias which findings count as high severity — the biggest lever on
|
|
47
|
+
false positives. No threat model → map the surface yourself first (Front 0 · Discovery).
|
|
48
|
+
|
|
49
|
+
## Checklist
|
|
50
|
+
- [ ] Stack and package ecosystem(s) detected, attack surface mapped
|
|
51
|
+
- [ ] Dependency audit run for each ecosystem
|
|
52
|
+
- [ ] Source→sink paths traced across the four vulnerability classes
|
|
53
|
+
- [ ] Configuration and secret leakage scanned
|
|
54
|
+
- [ ] Authorization matrix produced, unprotected sensitive endpoints searched for
|
|
55
|
+
- [ ] Code builds on a model (prompts, retrieval, memory, tools, sub-agents, MCP) → Front 5 run (`references/ai-agents.md`)
|
|
56
|
+
- [ ] Each candidate adversarially **verified** (N-verifier disprove pass); FALSE_POSITIVE / CANNOT_VERIFY separated out
|
|
57
|
+
- [ ] Severity **derived from preconditions × access** (not the scanner's category); verification ≠ severity
|
|
58
|
+
- [ ] Findings reported in severity order, no secret disclosed; ruled-out findings recorded too
|
|
59
|
+
- [ ] Fix options presented to the user
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## Fronts
|
|
64
|
+
|
|
65
|
+
Five review fronts — Discovery · Dependencies · Code (source→sink) · Configuration · Authorization: **`references/fronts.md`** (read the fronts you're scanning). A sixth, for code built on a model — prompts, retrieval, memory, tools, sub-agents, MCP — lives in **`references/ai-agents.md`**: read it only when that code is present.
|
|
66
|
+
|
|
67
|
+
When you fan out a sub-agent per front, how you prompt it decides recall — describe vulnerability **shapes** not a
|
|
68
|
+
checklist, scope each agent, and state that vulnerabilities exist: **`references/prompting.md`**.
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
## Verify before you report
|
|
73
|
+
|
|
74
|
+
Discovery is deliberately noisy (recall-biased); a **separate adversarial pass** disproves each candidate before it
|
|
75
|
+
reaches the report — the biggest lever on false positives. Run **N independent verifiers** per finding (each starts
|
|
76
|
+
from the code, not the summary; each hunts for why it's *wrong*), classify **TRUE_POSITIVE / FALSE_POSITIVE /
|
|
77
|
+
CANNOT_VERIFY**, then derive **severity from preconditions × access** (independently — "real" is not "critical").
|
|
78
|
+
The verifier procedure, the false-positive exclusion rules, the parseable verdict block, and the severity matrix
|
|
79
|
+
live in **`references/verify.md`**. Ruled-out findings are recorded, not silently dropped.
|
|
80
|
+
|
|
81
|
+
## Report
|
|
82
|
+
|
|
83
|
+
The severity scale, finding format, summary line, and the fix-presentation format live in **`references/reporting.md`**.
|
|
84
|
+
|
|
85
|
+
## Invariant rules
|
|
86
|
+
1. **Guides, does not assure** — it does not replace a professional audit; say so in the report.
|
|
87
|
+
2. **State coverage, not just findings** — every report ends with a `complete / partial / unknown` ledger over the
|
|
88
|
+
threat model's surfaces, each non-complete row carrying its reason. Without it, "no findings" and "never
|
|
89
|
+
looked" read identically to whoever acts on the report (`references/reporting.md`).
|
|
90
|
+
3. **Mask secrets** — only the first 4 + last 4 characters (`sk-p…i789`); never write the full secret.
|
|
91
|
+
4. **No automatic fix without approval** — even if "Fix everything" is chosen, first show what will change.
|
|
92
|
+
5. **Do not install tools without asking.**
|
|
93
|
+
6. **Preserve behavior** — a fix must not change functionality beyond closing the vulnerability.
|
|
94
|
+
7. **Stay local** — do not send code/data to an external service, do not cross the project boundary.
|
|
95
|
+
8. **Do not run the code under review to prove a finding** — no build, test, install or fixture run of it unless it
|
|
96
|
+
is isolated: no network, an empty environment, writes confined to a scratch directory, and hard limits on time
|
|
97
|
+
and resources. An install runs the target's own scripts; a test run executes whatever the target chose. If that
|
|
98
|
+
isolation is not available, the finding stays **CANNOT_VERIFY** with the exact local check that would settle
|
|
99
|
+
it. Reading source never needs any of this; executing it always does.
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
# Front 5 — AI, agent and MCP code
|
|
2
|
+
|
|
3
|
+
Read this front when the code under review builds on a model: it assembles prompts, retrieves context, stores
|
|
4
|
+
memory, exposes or calls tools, delegates to sub-agents, or speaks MCP. It is about the CODE around the model. For
|
|
5
|
+
testing whether a deployed agent's defences hold under attack, that is `red-team`.
|
|
6
|
+
|
|
7
|
+
## The rule that decides what counts
|
|
8
|
+
|
|
9
|
+
**Persuasive text is not a vulnerability. A missing deterministic control is.** A model can always be talked into
|
|
10
|
+
something; that alone proves nothing about the system. A finding here needs a boundary that code failed to hold:
|
|
11
|
+
content reached another principal's context, an action ran with authority the requester did not have, data was
|
|
12
|
+
disclosed to someone who could not read it, or a sink was driven that the attacker could not reach directly.
|
|
13
|
+
|
|
14
|
+
Three consequences follow, and they rule out most false positives in this front:
|
|
15
|
+
|
|
16
|
+
- **Instructions in a prompt guard nothing.** "The system prompt forbids it" is not a control. Count only what
|
|
17
|
+
a deterministic path enforces: authorization scoped to the resource, isolation, binding, constrained
|
|
18
|
+
credentials.
|
|
19
|
+
- **Everything the model reads or writes is untrusted input** — its own output, retrieved documents, memory, tool
|
|
20
|
+
descriptions, MCP responses. Point to the line of code that trusts it.
|
|
21
|
+
- **Using your own authority on purpose is not a defect.** A user who asks the agent to do something they are
|
|
22
|
+
allowed to do has not been attacked because a model carried it out.
|
|
23
|
+
|
|
24
|
+
Name, for every candidate: the attacker, the affected principal, the identity the action actually ran as, the
|
|
25
|
+
resource, the exact action, the authority used, and the result someone can observe.
|
|
26
|
+
|
|
27
|
+
## Where it breaks
|
|
28
|
+
|
|
29
|
+
### Context, retrieval and memory
|
|
30
|
+
- **Injected content reaching someone else's context.** Who can write each retrieved source — a document, page,
|
|
31
|
+
ticket, email, tool result — and whose session ends up reading it? Check that retrieval is scoped to the reader,
|
|
32
|
+
not just that the content looks benign.
|
|
33
|
+
- **Context bleeding across sessions or tenants.** History, embeddings, retrieval results and prompt caches keyed
|
|
34
|
+
too broadly. A tenant field on the record is not enforcement if one query path or one cache key leaves it out.
|
|
35
|
+
- **Memory that turns low-trust observations into durable instructions.** Content an attacker influenced, or a
|
|
36
|
+
model's own summary of it, written into memory that later steers another user or a privileged task. Who may
|
|
37
|
+
create, merge and delete memory, and does retrieval tell a user's preference apart from policy? A note a user
|
|
38
|
+
keeps for themself, read back only by that user, crosses no boundary.
|
|
39
|
+
- **Forged provenance.** Prompt assembly that lets untrusted text pass as a system message, an earlier turn, a tool
|
|
40
|
+
result or a memory record — string concatenation, untyped history, a caller-supplied role field. It matters when
|
|
41
|
+
the forged label changes a trust decision or unlocks a capability.
|
|
42
|
+
|
|
43
|
+
### Tools and actions
|
|
44
|
+
- **Model-written arguments reaching a sink unchecked.** A tool schema shapes input; it does not authorize a path,
|
|
45
|
+
a URL, a query or a resource id. Follow each argument from the decoded call to the SQL, shell, file or network
|
|
46
|
+
call, and look for the handler's own validation.
|
|
47
|
+
- **Confused deputy.** The agent acts with a broad service credential and the handler never re-checks whether the
|
|
48
|
+
requesting principal may touch that resource. Ask what that user could already do without the agent. One service
|
|
49
|
+
credential is fine when every call made with it is still scoped to the requester.
|
|
50
|
+
- **Approval that does not bind the action.** The user approves one described action, and execution can use
|
|
51
|
+
different arguments, a different target, a later turn, a retry or a resumed session. Or attacker content triggers
|
|
52
|
+
a side effect under the victim's valid authority that the victim never asked for — generic permission does not
|
|
53
|
+
make that intentional. The approval has to bind the normalized tool name, the complete argument object, the
|
|
54
|
+
requester and the target.
|
|
55
|
+
- **Schema and handler disagreeing.** Validation and the handler read one input two ways: an alias, a repeated or
|
|
56
|
+
unexpected key, a type that gets coerced, a free-form nested object, a value out of range. Validate again where a value
|
|
57
|
+
becomes a resource selector or a security option.
|
|
58
|
+
- **Delegated loops with no ceiling.** A bounded request that can enqueue unbounded spend, sends, mutations or
|
|
59
|
+
external calls — no per-request budget, no per-action check, no cancellation, no idempotency. Establish it from
|
|
60
|
+
the code and a locally bounded loop, never by exhausting a real service.
|
|
61
|
+
|
|
62
|
+
### MCP and sub-agents
|
|
63
|
+
- **Delegation that hands over everything.** A sub-agent or MCP call receives the whole session, its credentials,
|
|
64
|
+
its memory, rather than the least authority the task needs; or its result is trusted on return.
|
|
65
|
+
- **Identity by name instead of by connection.** Calls or results routed by a server name, tool name, request id,
|
|
66
|
+
resource URI or a model-chosen alias that an attacker can influence, instead of the authenticated connection and
|
|
67
|
+
the pending request. Can two servers claim one tool? Does a reconnect change the binding? Can one server's
|
|
68
|
+
response satisfy another's pending call?
|
|
69
|
+
- **Metadata treated as policy.** Whatever an MCP peer says about itself — tool text, resource details, suggested
|
|
70
|
+
prompts, schemas, completions — can steer the model and can never grant it anything. Find the allowlist, the server
|
|
71
|
+
identity check and the handler authorization that stay authoritative when the metadata says otherwise.
|
|
72
|
+
|
|
73
|
+
### Output and disclosure
|
|
74
|
+
- **Model output rendered unencoded.** Output reaching HTML, Markdown, a template, a URL or a command without that
|
|
75
|
+
sink's encoding. Whether a renderer outside the repository sanitizes it is a fact you do not have — that makes
|
|
76
|
+
the candidate CANNOT_VERIFY, not confirmed.
|
|
77
|
+
- **Secrets extractable from the assembled context.** Credentials, another user's data or access-granting policy
|
|
78
|
+
values placed in context that user-influenced output can reveal. Leaking generic instructions crosses no
|
|
79
|
+
boundary and is not a finding.
|
|
80
|
+
|
|
81
|
+
## How to hunt it
|
|
82
|
+
|
|
83
|
+
Before opening single files, make four inventories: the identities code runs under, the capabilities it holds, the
|
|
84
|
+
context and memory stores something can write into, and the destinations output reaches. Then trace each request
|
|
85
|
+
from the one who made it to the power finally used on its behalf. From each tool with side effects, walk back
|
|
86
|
+
through dispatch, schema, confirmation, context, retrieval and ingestion; from each memory read, walk back to
|
|
87
|
+
everyone who writes there. One action can arrive directly, from a queue, as a retry, a resumed run, a batch or
|
|
88
|
+
through a delegate — the deciding check belongs after the arguments are fixed and before anything happens, on every
|
|
89
|
+
one of those routes.
|
|
90
|
+
|
|
91
|
+
## Before a finding here leaves the verifier
|
|
92
|
+
|
|
93
|
+
Apply `verify.md` as usual, plus:
|
|
94
|
+
|
|
95
|
+
1. It says which boundary was crossed and what anyone can see afterwards: who attacks, who (or which shared
|
|
96
|
+
resource) is harmed, the identity the action ran as, the target, and the action or disclosure nobody authorized
|
|
97
|
+
or asked for.
|
|
98
|
+
2. A confused-deputy claim shows two things: the handler never checks the requester against the resource, AND the
|
|
99
|
+
attacker has no way to do the same through the product itself. An action-binding claim shows instead that
|
|
100
|
+
content the attacker controlled set off an action, carried out with the victim's authority, that the victim did
|
|
101
|
+
not mean to request or approve.
|
|
102
|
+
3. A memory or retrieval claim shows both halves: where the attacker writes, and where another principal or a
|
|
103
|
+
privileged decision later reads. A shared record that nothing reads across a boundary does not qualify.
|
|
104
|
+
4. An MCP identity claim looks at four things: which connection is authenticated, how a response is matched to its
|
|
105
|
+
request, how tool names are namespaced, and which credential is actually used. When the answer depends on
|
|
106
|
+
routing done by an outside server or the deployment, the verdict is CANNOT_VERIFY, naming the one fact that
|
|
107
|
+
would decide it.
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# Review fronts
|
|
2
|
+
|
|
3
|
+
The scan applies the **source → gate → sink** model on these fronts. Read the fronts you're scanning.
|
|
4
|
+
|
|
5
|
+
## Front 0 — Discovery (map the surface)
|
|
6
|
+
|
|
7
|
+
The rest of the audit rests on the map produced here.
|
|
8
|
+
|
|
9
|
+
1. Scan for manifest/build files at the root and in subdirectories: `package.json`, `requirements.txt`, `pyproject.toml`, `go.mod`, `Cargo.toml`, `Gemfile`, `pom.xml`, `build.gradle`, `composer.json`, `*.csproj`, `*.sln`, `mix.exs`. In a monorepo, note each ecosystem separately.
|
|
10
|
+
2. Derive the framework and runtime from the manifest.
|
|
11
|
+
3. **List the attack surface:** all the points where user input enters the system (the "Source" list above). These are the starting points for the following fronts.
|
|
12
|
+
4. Run a **web search** for the detected framework — tooling and patterns change often: `"[framework] security best practices"`, `"[framework] common vulnerabilities"`, the current `"OWASP Top 10"` list.
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## Front 1 — Dependencies
|
|
17
|
+
|
|
18
|
+
Known vulnerabilities in third-party packages are the cheapest path to most breaches.
|
|
19
|
+
|
|
20
|
+
1. Run the audit command that suits the ecosystem (if it is not installed, suggest installing it, do not install it yourself):
|
|
21
|
+
|
|
22
|
+
| Ecosystem | Command |
|
|
23
|
+
|---|---|
|
|
24
|
+
| Node | `npm audit` · `pnpm audit` · `yarn npm audit` |
|
|
25
|
+
| Python | `pip-audit` |
|
|
26
|
+
| .NET | `dotnet list package --vulnerable --include-transitive` |
|
|
27
|
+
| Rust | `cargo audit` |
|
|
28
|
+
| Go | `govulncheck ./...` |
|
|
29
|
+
| Ruby | `bundle audit` |
|
|
30
|
+
| Java | OWASP Dependency-Check |
|
|
31
|
+
|
|
32
|
+
2. If you are unsure of the correct/current command, verify it on the web; where possible, get machine-readable output with `--json`.
|
|
33
|
+
3. Extract from each finding: package · installed version · vulnerability id (CVE/GHSA) · severity · fixed version.
|
|
34
|
+
4. In a monorepo, audit each sub-project separately. The results go into the report (below).
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
## Front 2 — Code (source→sink tracing)
|
|
39
|
+
|
|
40
|
+
Start from every entry point on the attack surface, trace the input to where it is used, and check whether there is a gate in between. The table below is the **minimum** coverage — if you see another sink during the scan, add it.
|
|
41
|
+
|
|
42
|
+
| Sink class | Vulnerability | What the "gate" should be |
|
|
43
|
+
|---|---|---|
|
|
44
|
+
| **Query** | SQL / NoSQL / LDAP / ORM raw query injection | Parameterized query; no string concatenation |
|
|
45
|
+
| **Command** | Shell/OS command injection | Argument array; no passing a string to the shell |
|
|
46
|
+
| **Render** | XSS (input in HTML without escaping), SSTI (input as template code) | Context-appropriate escaping; input as data, not code |
|
|
47
|
+
| **Path** | Path traversal, unrestricted file upload | Path allowlist/normalization; type-size-content validation |
|
|
48
|
+
| **Object** | Insecure deserialization, mass assignment | Safe format; field allowlist |
|
|
49
|
+
| **Expression** | Expression/eval injection, `eval`/`exec` | Never evaluate input as code |
|
|
50
|
+
|
|
51
|
+
**On state-changing + browser-originating endpoints, additionally:**
|
|
52
|
+
- **CSRF** — does the state-changing endpoint have token protection?
|
|
53
|
+
- **CORS** — is there a wildcard origin with credentials, or an origin reflected without validation?
|
|
54
|
+
|
|
55
|
+
**Secret leakage — hardcoded credentials in the source:** search for these patterns, excluding `.env.example` and test fixtures:
|
|
56
|
+
```
|
|
57
|
+
(password|api[_-]?key|secret|token)\s*[:=]\s*["'][^"']+["']
|
|
58
|
+
-----BEGIN (RSA |EC |DSA )?PRIVATE KEY-----
|
|
59
|
+
```
|
|
60
|
+
Also: unmasked password/token/PII in logs; a stack trace or internal path leaked to the user.
|
|
61
|
+
|
|
62
|
+
**Tracing method:** read the route/controller files (input enters here) → grep for dangerous calls (`eval`, `exec`, string-SQL) → check whether the middleware/interceptor has auth·CSRF·rate-limit → check whether the data model has mass-assignment protection. Run a web search for language-specific dangerous functions.
|
|
63
|
+
|
|
64
|
+
**Record for each finding:** file:line · vulnerability · what an attacker could do (impact) · fix specific to this code.
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
|
|
68
|
+
## Front 3 — Configuration
|
|
69
|
+
|
|
70
|
+
**Is a secret in the repo:**
|
|
71
|
+
```bash
|
|
72
|
+
git ls-files --error-unmatch .env 2>/dev/null && echo "WARNING: .env is tracked"
|
|
73
|
+
```
|
|
74
|
+
If it's a finding, CRITICAL: `git rm --cached .env` → add to `.gitignore` → **rotate the exposed secrets**.
|
|
75
|
+
|
|
76
|
+
**Debug on in prod:** search for the framework-specific flag with `"[framework] debug mode production"`. Signs: an open debug flag, an error page showing a stack trace, a published source map, accessible development endpoints.
|
|
77
|
+
|
|
78
|
+
**Insecure transport:** hardcoded `http://` addresses (API, webhook, OAuth redirect, CDN) that should be `https`, and cookies without `Secure`. Exclude `localhost`/`127.0.0.1`/`0.0.0.0`.
|
|
79
|
+
|
|
80
|
+
**Missing security headers:** CSP, HSTS, X-Frame-Options, X-Content-Type-Options.
|
|
81
|
+
|
|
82
|
+
---
|
|
83
|
+
|
|
84
|
+
## Front 4 — Authorization (auth-z)
|
|
85
|
+
|
|
86
|
+
Access-control flaws are the class most often missed in a scan, because the code appears to "work." Mandatory if the project includes admin/dashboard or role-based access:
|
|
87
|
+
|
|
88
|
+
1. Enumerate all roles/levels (admin, moderator, user, guest…).
|
|
89
|
+
2. Write all routes/endpoints and the **required** permission level into a matrix.
|
|
90
|
+
3. Verify that each protected route actually checks — "being logged in" is not authorization.
|
|
91
|
+
4. Look for endpoints that should be protected but aren't; try whether there is an admin endpoint reachable by guessing the URL.
|
|
92
|
+
5. Verify that authorization is enforced both at the controller **and** at the data-access layer (IDOR: object reference without ownership validation).
|
|
93
|
+
6. Privilege escalation: can a user change their own role/permission? Does a sensitive admin action require extra protection (re-auth/2FA)?
|
|
94
|
+
7. **JWT:** `alg: none` / algorithm confusion, secret embedded in code, missing expiration.
|
|
95
|
+
|
|
96
|
+
## Front 5 — AI, agent and MCP code
|
|
97
|
+
|
|
98
|
+
Only when the code under review builds on a model. The classes, the rule that decides what counts, and the extra
|
|
99
|
+
checks a finding must pass before it leaves the verifier are in **`ai-agents.md`**.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Prompting the review sub-agents (defensive security)
|
|
2
|
+
|
|
3
|
+
When the scan fans out a sub-agent per front, *how* you prompt it decides recall. Field-tested rules — these also
|
|
4
|
+
apply to `red-team` and any agent you point at a codebase to find bugs:
|
|
5
|
+
|
|
6
|
+
1. **Give the model room.** A high-level task beats prescriptive scaffolding. Long instructions, staged checklists,
|
|
7
|
+
and piles of reference material tend to backfire — the model pattern-matches your scaffold instead of reasoning
|
|
8
|
+
about the code.
|
|
9
|
+
2. **Describe vulnerability *shapes*, not a checklist.** Enumerating bug types ("find SQLi, XSS, and CSRF")
|
|
10
|
+
**worsens** recall — the model stops at your list. Describe the structural property instead: *"attacker input
|
|
11
|
+
that alters the syntactic structure of an interpreted string — SQL, shell, HTML, a template — or crosses a trust
|
|
12
|
+
boundary without a gate."*
|
|
13
|
+
3. **Scope each agent.** Tell it precisely which slice of the surface to search **and which classes you don't care
|
|
14
|
+
about**, so parallel agents don't all converge on the same easy findings. (A `threat-model` gives you this scope.)
|
|
15
|
+
4. **Frame verification adversarially.** "Actively look for reasons this finding is *wrong*" beats "check for
|
|
16
|
+
protections" — the whole disprove pass lives in `references/verify.md`.
|
|
17
|
+
5. **State that vulnerabilities exist.** "Act like a pentester" is weak. Strong: *"Produce a findings report with a
|
|
18
|
+
PoC for each Critical/High; assume there ARE vulnerabilities here; keep going until you've covered the surface."*
|
|
19
|
+
6. **Share mitigations the code doesn't show** (a WAF rule, a gateway auth check, a platform default) up front — it
|
|
20
|
+
saves the reviewer from re-finding a path that is already closed one layer up (a common false positive).
|
|
21
|
+
7. **Prompt-level guardrails are defense-in-depth only.** Real safety is human approval + the tool-level gates
|
|
22
|
+
(§4.4/§4.5), never a sentence in a prompt. Don't rely on "please don't exploit this" — rely on the gate.
|
|
23
|
+
8. **The sub-agent's final report is all the caller reads** — put every finding and its coverage in it.
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# Reporting & fix presentation
|
|
2
|
+
|
|
3
|
+
## Report
|
|
4
|
+
|
|
5
|
+
**Severity scale:**
|
|
6
|
+
|
|
7
|
+
| Level | Meaning |
|
|
8
|
+
|---|---|
|
|
9
|
+
| CRITICAL | Directly exploitable — urgent (SQL injection, hardcoded secret in prod) |
|
|
10
|
+
| HIGH | Serious — close before deploy (XSS, command injection, known CVE) |
|
|
11
|
+
| MEDIUM | Defense gap — soon (missing CSRF, loose CORS) |
|
|
12
|
+
| LOW | Best-practice violation — when convenient (debug mode, missing header) |
|
|
13
|
+
| INFO | Observation — no immediate risk |
|
|
14
|
+
|
|
15
|
+
**Finding format** (ranked by severity):
|
|
16
|
+
```
|
|
17
|
+
[CRITICAL] SQL Injection — src/api/users.ts:42
|
|
18
|
+
Vuln : Input concatenated directly into the SQL query
|
|
19
|
+
Impact: DB can be read / modified / deleted
|
|
20
|
+
Fix : Use a parameterized query
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
**Summary line:**
|
|
24
|
+
```
|
|
25
|
+
=== Security Scan Summary ===
|
|
26
|
+
CRITICAL: X · HIGH: X · MEDIUM: X · LOW: X · INFO: X · Total: X
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
## Coverage — say what you did NOT look at
|
|
30
|
+
|
|
31
|
+
A report with no findings is read as "nothing is wrong". It can equally mean "that surface was never opened",
|
|
32
|
+
and the two are indistinguishable to the person acting on it — who then ships believing an audit happened. The
|
|
33
|
+
`CANNOT_VERIFY` verdict already handles this for an individual finding; coverage handles it for the audit.
|
|
34
|
+
|
|
35
|
+
Every report ends with a coverage ledger, one row per surface from the threat model ([[threat-model]]):
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
=== Coverage ===
|
|
39
|
+
complete auth/session handling — read every route + middleware
|
|
40
|
+
complete SQL data access — grepped all query construction sites
|
|
41
|
+
partial file upload — read the handler; storage backend config not available
|
|
42
|
+
unknown payment webhook — third-party signature logic, no source access
|
|
43
|
+
unknown admin client (Blazor) — out of scope this pass, agreed with the user
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Rules:
|
|
47
|
+
- **`complete` is a claim, not a hope.** Use it only where you can name what you read or ran. If you sampled,
|
|
48
|
+
it is `partial`.
|
|
49
|
+
- Every `partial` and `unknown` carries **why** — no access, out of scope, no time, third-party. A bare
|
|
50
|
+
`unknown` is the same silence the ledger exists to remove.
|
|
51
|
+
- `unknown` on a surface the threat model calls high-risk is itself a finding: report it at the severity that
|
|
52
|
+
surface would carry, not as a footnote.
|
|
53
|
+
- The ledger is written **before** the fixes are proposed. Written afterwards, it gets trimmed to match the
|
|
54
|
+
work that happened to be done.
|
|
55
|
+
- "Zero findings, coverage complete" is a strong claim and rare. "Zero findings, half the surface unknown" is
|
|
56
|
+
an honest and much more common one.
|
|
57
|
+
|
|
58
|
+
### When a previous report exists
|
|
59
|
+
Read it before planning, and let it shape priority — never coverage.
|
|
60
|
+
- A previous **TRUE_POSITIVE** carries into this report only if the code on its path is unchanged; the verifier
|
|
61
|
+
re-checks it like any other candidate. Changed code makes it a fresh candidate, not an inherited one.
|
|
62
|
+
- A previous **FALSE_POSITIVE** rules out that exact claim on that exact unchanged code — nothing more. The surface
|
|
63
|
+
it sat on is still reviewed.
|
|
64
|
+
- A previous **CANNOT_VERIFY** is current work, unless the fact it waited on is still unavailable; then it carries
|
|
65
|
+
with that same named blocker.
|
|
66
|
+
- A previous `partial` or `unknown` row is never inherited as `complete`, and a previous narrow or quick pass is
|
|
67
|
+
never read as "the rest was fine". Say in this report which rows came from the earlier one and why they still hold.
|
|
68
|
+
|
|
69
|
+
## Fix presentation
|
|
70
|
+
```
|
|
71
|
+
How shall we proceed?
|
|
72
|
+
1. Fix everything 2. CRITICAL+HIGH only
|
|
73
|
+
3. Approve one by one 4. Manual (no changes)
|
|
74
|
+
```
|
|
75
|
+
For each fix: preview the diff → wait for approval → (if a dependency) upgrade command + breaking-change note → re-run the relevant check.
|
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
# Verify before you report — the adversarial pass
|
|
2
|
+
|
|
3
|
+
Discovery and verification want opposite things. **Discovery is recall-biased** — when unsure whether something is
|
|
4
|
+
real, surface it (a low-confidence finding beats a missed bug). **Verification is precision-biased** — a separate,
|
|
5
|
+
adversarial pass that tries to *disprove* each finding. Keeping them separate is the single biggest lever on false
|
|
6
|
+
positives. (This is the same two-stage-verdict discipline `crew-code-review` uses — reused here for security findings.)
|
|
7
|
+
|
|
8
|
+
Run this pass on every candidate before it reaches the report.
|
|
9
|
+
|
|
10
|
+
## The verifier stance
|
|
11
|
+
For each finding, run **N independent verifiers** (default **3**; 1 for a quick scan, 5 for a release/high-stakes
|
|
12
|
+
audit). Each verifier:
|
|
13
|
+
- **Starts from the code at the cited `file:line`, not the finding's summary.** If you start from the summary you
|
|
14
|
+
inherit its misreading.
|
|
15
|
+
- Is stanced **"find any reason this finding is wrong."** Actively hunt for why it is *not* exploitable.
|
|
16
|
+
- Is **blind to the other verifiers' reasoning** — shared context propagates the same blind spot.
|
|
17
|
+
|
|
18
|
+
### "Independent" means a separate context, or it means nothing
|
|
19
|
+
Three passes inside one context window are not three verifiers. The second pass has already read the first one's
|
|
20
|
+
reasoning; the blindness above is then a wish, not a property, and the agreement it produces is one argument
|
|
21
|
+
counted three times. So:
|
|
22
|
+
- **Each verifier is its own subagent** (`Agent`, one call per verifier, run in parallel). A subagent has its own
|
|
23
|
+
context window: it cannot see what the others concluded, which is the whole point. Read-only work, so no
|
|
24
|
+
`worktree` is needed — nothing is being written.
|
|
25
|
+
- **Give it the finding and the file:line, never your reasoning.** Not "I think X is exploitable because Y" —
|
|
26
|
+
just "claim: X at file:line. Try to refute it."
|
|
27
|
+
- **If you did not isolate, say so.** Write `verifiers: 1 (single pass, not isolated)` in the verdict block. A
|
|
28
|
+
single honest pass is useful; three passes labelled as independent when they were not is worse than one,
|
|
29
|
+
because it launders confidence.
|
|
30
|
+
|
|
31
|
+
**Cost gate, because isolation is not free** — each subagent re-pays for its own context (see `token-budget`), so
|
|
32
|
+
N×findings gets expensive fast. Isolate by stake, not by default:
|
|
33
|
+
- **critical / high severity, or anything heading for a release** → isolate, N=3 (N=5 for a release audit).
|
|
34
|
+
- **medium** → isolate, N=1. One genuinely independent pass beats three entangled ones.
|
|
35
|
+
- **low** → single inline pass, labelled as such. Do not spend a subagent proving a `low`.
|
|
36
|
+
|
|
37
|
+
### The verifiers must not be three copies of one method
|
|
38
|
+
Independent is still not uncorrelated. Three verifiers running the same four steps fail in the same place, and
|
|
39
|
+
their agreement then reads as confirmation when it is only repetition. Assign each a **different attack**:
|
|
40
|
+
- **Reachability** — can untrusted input actually arrive here? Quote the first call-site as evidence.
|
|
41
|
+
- **Protection** — does something on every path already stop it? Try to break each guard.
|
|
42
|
+
- **Reproduction** — what input, in what state, actually triggers it? If you cannot name one, that is
|
|
43
|
+
`CANNOT_VERIFY`, not agreement with the others.
|
|
44
|
+
|
|
45
|
+
At N=5 add **exclusion rules** (does this match a known FALSE_POSITIVE class below?) and **blast radius** (if it
|
|
46
|
+
is real, what does it actually reach?). Record which lens produced which verdict — a verdict whose lens is
|
|
47
|
+
unnamed cannot be weighed.
|
|
48
|
+
|
|
49
|
+
### Unanimity is a signal to check, not a verdict to trust
|
|
50
|
+
If every verifier returns the same verdict **for the same reason**, that is one argument counted N times. Before
|
|
51
|
+
accepting it, run one counterfactual pass: *"assume this verdict is wrong — what would have to be true, and what
|
|
52
|
+
evidence would show it?"* Record the answer beside the verdict. Unanimity with N distinct reasons is strong;
|
|
53
|
+
unanimity with one reason is a monoculture, and it is exactly how a whole panel misses the same guard.
|
|
54
|
+
|
|
55
|
+
## The 4-step verify procedure (each step kills a specific false-positive class)
|
|
56
|
+
1. **Read the cited code yourself** — do not trust the description.
|
|
57
|
+
2. **Trace reachability backward from the sink.** Can untrusted input actually reach it? **Quote the first
|
|
58
|
+
call-site `file:line`** as evidence. *Unreachable code is the single largest source of false positives* — a
|
|
59
|
+
plausible-sounding chain is not enough.
|
|
60
|
+
3. **Hunt for protections** on every path: validation, parameterization, auto-escaping, type/length bounds, an
|
|
61
|
+
auth/scope gate, or the code being dead/test-only.
|
|
62
|
+
4. **Stress-test each protection on every path** — a guard that holds on one route but not another is not a guard.
|
|
63
|
+
|
|
64
|
+
## Verdict — three outcomes, never a forced binary
|
|
65
|
+
- **TRUE_POSITIVE** — requires ALL of: reachable + protections insufficient + exploitation feasible.
|
|
66
|
+
- **FALSE_POSITIVE** — fails one of the above, or matches an exclusion rule below.
|
|
67
|
+
- **CANNOT_VERIFY** — static reasoning hit its limit (needs a running system / data you don't have). Report it
|
|
68
|
+
honestly as `needs_manual_test`; never inflate uncertainty into a confident verdict.
|
|
69
|
+
- It carries **no severity**. It is a specific, source-grounded hypothesis that one missing fact blocks — not a
|
|
70
|
+
low-confidence confirmed vulnerability — and a severity on it would be read as a rating it has not earned.
|
|
71
|
+
- Name the missing fact exactly, and the check that would settle it.
|
|
72
|
+
- Controls that live outside the repository — a proxy, a WAF, provider settings, identity policy, network
|
|
73
|
+
topology, how a renderer sanitizes — are real controls. When a verdict depends on one that the source does not
|
|
74
|
+
show, do not assume it is **present**, and do not assume it is **absent**: that is what CANNOT_VERIFY is for.
|
|
75
|
+
|
|
76
|
+
"I couldn't write a working PoC" is **weak** evidence of non-exploitability — do not downgrade on that alone.
|
|
77
|
+
|
|
78
|
+
## Exclusion rules — FALSE_POSITIVE even if technically accurate
|
|
79
|
+
1. Volumetric/rate DoS with no amplification (just "many requests").
|
|
80
|
+
2. Test, fixture, example, or provably dead/unreachable code.
|
|
81
|
+
3. Intended, documented design (a stated, accepted risk).
|
|
82
|
+
4. Memory-safety issue in a memory-safe language/runtime.
|
|
83
|
+
5. Input that only a trusted operator controls (a local CLI flag, an ops-only env var).
|
|
84
|
+
6. "Outdated dependency" with no reachable vulnerable call path.
|
|
85
|
+
7. Weak randomness used somewhere non-security (a cache key, a test seed).
|
|
86
|
+
8. XSS auto-neutralised by the framework's default escaping.
|
|
87
|
+
9. Traversal blocked one layer up (a gateway/service the reasoning didn't read).
|
|
88
|
+
10. A theoretical-only TOCTOU with no realistic race window.
|
|
89
|
+
11. "Missing hardening" with no concrete exploit (a nice-to-have header).
|
|
90
|
+
12. Low-impact nuisance with no security consequence.
|
|
91
|
+
|
|
92
|
+
Add project-specific rules as you confirm them.
|
|
93
|
+
|
|
94
|
+
## Structured verdict block (one per finding, parseable)
|
|
95
|
+
```
|
|
96
|
+
VERDICT: TRUE_POSITIVE | FALSE_POSITIVE | CANNOT_VERIFY
|
|
97
|
+
CONFIDENCE: 0-10
|
|
98
|
+
VERIFIERS: <N, isolated | N, single pass not isolated>
|
|
99
|
+
LENSES: <reachability, protection, reproduction — which ones actually ran>
|
|
100
|
+
AGREEMENT: <unanimous-same-reason | unanimous-distinct-reasons | split N-M>
|
|
101
|
+
REACHABILITY: <first untrusted call-site file:line, or "unreachable">
|
|
102
|
+
EXCLUSION_RULE: <number, or none>
|
|
103
|
+
RATIONALE: <cites concrete file:line, not prose>
|
|
104
|
+
```
|
|
105
|
+
`VERIFIERS` and `AGREEMENT` exist so the reader can discount the verdict without re-doing the work. "3, single
|
|
106
|
+
pass not isolated" and "unanimous-same-reason" together mean one argument wearing three hats — still worth
|
|
107
|
+
reporting, not worth treating as three. Leaving these fields out is how a weak verdict passes for a strong one.
|
|
108
|
+
|
|
109
|
+
## Tally
|
|
110
|
+
Verdict = majority of the N verifiers; confidence = mean of the agreeing votes. **Tie → the operator's noise
|
|
111
|
+
tolerance:** precision → drop; recall → keep as `needs_manual_test`; ask → one batched question. Never silently
|
|
112
|
+
drop a finding — a dropped finding is recorded (with its verdict) in the report's "Ruled out" section.
|
|
113
|
+
|
|
114
|
+
## Severity is a SEPARATE judgment from verification
|
|
115
|
+
"This is real" must not inflate into "this is critical." Derive severity from **preconditions × access level**, not
|
|
116
|
+
from the vulnerability category:
|
|
117
|
+
|
|
118
|
+
| access ↓ / preconditions → | 0 preconditions | 1–2 | 3+ |
|
|
119
|
+
|---|---|---|---|
|
|
120
|
+
| **unauthenticated / remote** | CRITICAL–HIGH | MEDIUM | LOW |
|
|
121
|
+
| **authenticated** | MEDIUM | MEDIUM | LOW |
|
|
122
|
+
| **local / operator** | LOW | LOW | INFO |
|
|
123
|
+
|
|
124
|
+
- Take the **lower** of the two axes. If the precondition list has **3+ items, HIGH is almost certainly wrong.**
|
|
125
|
+
- **HIGH or MEDIUM? Ask one question:** does the demonstrated result fully **defeat** an explicit control for an
|
|
126
|
+
action with real consequences, or only **weaken** it? Defeat is HIGH territory; weaken is MEDIUM at most. If you
|
|
127
|
+
cannot state the concrete damage in a sentence, the severity is lower than it feels.
|
|
128
|
+
- Severity never exceeds the impact actually demonstrated.
|
|
129
|
+
- A `threat-model` match may raise severity by **at most one step** — never two (a stated threat can't re-inflate a LOW to HIGH).
|
|
130
|
+
- Record `severity_alignment` (**-5..+5**): judged from the stance of a reviewer who has seen two hundred inflated
|
|
131
|
+
scanner findings this week — how over- or under-stated is the raw claim? Negative = deflate it.
|