@fro.bot/systematic 3.2.3 → 3.2.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/agents/document-review/adversarial-document-reviewer.md +5 -3
- package/agents/document-review/coherence-reviewer.md +6 -4
- package/agents/document-review/design-lens-reviewer.md +5 -3
- package/agents/document-review/feasibility-reviewer.md +5 -3
- package/agents/document-review/product-lens-reviewer.md +5 -3
- package/agents/document-review/scope-guardian-reviewer.md +5 -3
- package/agents/document-review/security-lens-reviewer.md +5 -3
- package/package.json +1 -1
- package/skills/document-review/SKILL.md +7 -6
- package/skills/document-review/references/findings-schema.json +5 -6
- package/skills/document-review/references/review-output-template.md +48 -34
- package/skills/document-review/references/subagent-template.md +11 -11
- package/skills/document-review/references/synthesis-and-presentation.md +3 -3
|
@@ -73,9 +73,11 @@ Probe whether the document considered the obvious alternatives and whether the c
|
|
|
73
73
|
|
|
74
74
|
## Confidence calibration
|
|
75
75
|
|
|
76
|
-
- **
|
|
77
|
-
- **
|
|
78
|
-
- **
|
|
76
|
+
- **0:** The challenge is a false positive or a pre-existing issue. Suppress it.
|
|
77
|
+
- **25:** The premise or assumption might be wrong, but the available document and codebase evidence cannot verify it. Suppress it.
|
|
78
|
+
- **50:** The challenge is verified, but it is an advisory or low-impact concern that does not materially threaten the decision. Return it as FYI only.
|
|
79
|
+
- **75:** You have double-checked the premise or decision against the document (and relevant codebase evidence where available), can construct a concrete failure scenario, and the consequence directly affects correctness in practice. This is actionable.
|
|
80
|
+
- **100:** The document directly confirms the premise failure or missing assumption, and the concrete failure scenario will occur frequently on the plan's normal path. Reserve this exceptional actionable anchor for direct evidence, not a persuasive counterargument; it is the only anchor eligible for a silent fix.
|
|
79
81
|
|
|
80
82
|
## What you don't flag
|
|
81
83
|
|
|
@@ -14,7 +14,7 @@ You are a technical editor reading for internal consistency. You don't evaluate
|
|
|
14
14
|
|
|
15
15
|
**Terminology drift** -- same concept called different names in different sections ("pipeline" / "workflow" / "process" for the same thing), or same term meaning different things in different places. The test is whether a reader could be confused, not whether the author used identical words every time.
|
|
16
16
|
|
|
17
|
-
**Structural issues** -- forward references to things never defined, sections that depend on context they don't establish, phased approaches where later phases depend on deliverables earlier phases don't mention. Also: requirements lists that span multiple distinct concerns without grouping headers. When requirements cover different topics (e.g., packaging, migration, contributor workflow), a flat list hinders comprehension for humans and agents. Flag with `autofix_class:
|
|
17
|
+
**Structural issues** -- forward references to things never defined, sections that depend on context they don't establish, phased approaches where later phases depend on deliverables earlier phases don't mention. Also: requirements lists that span multiple distinct concerns without grouping headers. When requirements cover different topics (e.g., packaging, migration, contributor workflow), a flat list hinders comprehension for humans and agents. Flag with `autofix_class: manual` when grouping requires a judgment call about the document's organization, keeping original R# IDs.
|
|
18
18
|
|
|
19
19
|
**Genuine ambiguity** -- statements two careful readers would interpret differently. Common sources: quantifiers without bounds, conditional logic without exhaustive cases, lists that might be exhaustive or illustrative, passive voice hiding responsibility, temporal ambiguity ("after the migration" -- starts? completes? verified?).
|
|
20
20
|
|
|
@@ -24,9 +24,11 @@ You are a technical editor reading for internal consistency. You don't evaluate
|
|
|
24
24
|
|
|
25
25
|
## Confidence calibration
|
|
26
26
|
|
|
27
|
-
- **
|
|
28
|
-
- **
|
|
29
|
-
- **
|
|
27
|
+
- **0:** The apparent inconsistency is a false positive or a pre-existing issue. Suppress it.
|
|
28
|
+
- **25:** The wording might be inconsistent, but the document does not provide enough evidence to verify that readers would diverge. Suppress it.
|
|
29
|
+
- **50:** The inconsistency is verified, but it is an advisory terminology or structural issue that is unlikely to affect implementation. Return it as FYI only.
|
|
30
|
+
- **75:** Two passages or a concrete reference comparison have been double-checked and would cause implementers to diverge or produce an incorrect interpretation in practice. This is actionable.
|
|
31
|
+
- **100:** Directly contradictory passages or a broken reference confirm an inconsistency that will recur frequently wherever the affected instruction is used. Reserve this exceptional anchor for direct textual evidence; it is the only anchor eligible for a silent fix.
|
|
30
32
|
|
|
31
33
|
## What you don't flag
|
|
32
34
|
|
|
@@ -35,9 +35,11 @@ Explain what's missing: the functional design thinking that makes the interface
|
|
|
35
35
|
|
|
36
36
|
## Confidence calibration
|
|
37
37
|
|
|
38
|
-
- **
|
|
39
|
-
- **
|
|
40
|
-
- **
|
|
38
|
+
- **0:** The design concern is a false positive or a pre-existing issue. Suppress it.
|
|
39
|
+
- **25:** A state, flow, or interaction might be missing, but the document does not let you verify the gap. Suppress it.
|
|
40
|
+
- **50:** The gap is verified, but it is advisory or low-impact and a reasonable implementation can proceed without resolving it. Return it as FYI only.
|
|
41
|
+
- **75:** You have double-checked a specific missing state, flow, or interaction against the document, and it will cause a practical implementation or user-flow failure. This is actionable.
|
|
42
|
+
- **100:** The document directly confirms a missing state or flow on a normal, frequently encountered interaction path, and the omission will repeatedly cause users to fail or implementers to block. Reserve this exceptional anchor for direct evidence; it is the only anchor eligible for a silent fix.
|
|
41
43
|
|
|
42
44
|
## What you don't flag
|
|
43
45
|
|
|
@@ -28,9 +28,11 @@ Apply each check only when relevant. Silence is only a finding when the gap woul
|
|
|
28
28
|
|
|
29
29
|
## Confidence calibration
|
|
30
30
|
|
|
31
|
-
- **
|
|
32
|
-
- **
|
|
33
|
-
- **
|
|
31
|
+
- **0:** The feasibility concern is a false positive or a pre-existing issue. Suppress it.
|
|
32
|
+
- **25:** The constraint or failure path might exist, but available document and codebase evidence cannot verify it. Suppress it.
|
|
33
|
+
- **50:** The constraint is verified, but it is an advisory or low-impact implementation concern that does not block the plan. Return it as FYI only.
|
|
34
|
+
- **75:** You have double-checked a concrete stack constraint, dependency, data-flow path, or migration condition and it will hit in practice, directly blocking correctness or implementation. This is actionable.
|
|
35
|
+
- **100:** Direct evidence from the stated stack, existing code, or an explicit plan constraint confirms that a normal path will fail frequently. Reserve this exceptional anchor for directly demonstrated, recurring incompatibility; it is the only anchor eligible for a silent fix.
|
|
34
36
|
|
|
35
37
|
## What you don't flag
|
|
36
38
|
|
|
@@ -59,9 +59,11 @@ If priority tiers exist: do assignments match stated goals? Are must-haves truly
|
|
|
59
59
|
|
|
60
60
|
## Confidence calibration
|
|
61
61
|
|
|
62
|
-
- **
|
|
63
|
-
- **
|
|
64
|
-
- **
|
|
62
|
+
- **0:** The product concern is a false positive or a pre-existing issue. Suppress it.
|
|
63
|
+
- **25:** The strategic concern might be real, but business context or evidence needed to verify it is absent. Suppress it.
|
|
64
|
+
- **50:** The misalignment is verified from the document, but it is an advisory, low-impact, or opportunity-cost observation. Return it as FYI only.
|
|
65
|
+
- **75:** You have double-checked the stated goal against the proposed work and the mismatch directly affects the likely outcome in practice. This is actionable. Qualitative strategic critiques top out here unless the document supplies direct contradiction or a hard boundary.
|
|
66
|
+
- **100:** Use this exceptional anchor only when the document directly contradicts its own stated goal or violates its own hard quantitative boundary, and the contradiction will occur frequently if followed. It is the only anchor eligible for a silent fix; do not use it for qualitative product judgment.
|
|
65
67
|
|
|
66
68
|
## What you don't flag
|
|
67
69
|
|
|
@@ -42,9 +42,11 @@ With AI-assisted implementation, the cost gap between shortcuts and complete sol
|
|
|
42
42
|
|
|
43
43
|
## Confidence calibration
|
|
44
44
|
|
|
45
|
-
- **
|
|
46
|
-
- **
|
|
47
|
-
- **
|
|
45
|
+
- **0:** The scope concern is a false positive or a pre-existing issue. Suppress it.
|
|
46
|
+
- **25:** The scope or complexity concern might be real, but the document does not provide enough evidence to verify it. Suppress it.
|
|
47
|
+
- **50:** The mismatch or complexity is verified, but it is advisory or low-impact and does not materially threaten delivery. Return it as FYI only.
|
|
48
|
+
- **75:** You have double-checked a stated goal against a scope item, abstraction, or priority boundary and the mismatch will directly affect right-sizing or delivery in practice. This is actionable. Qualitative scope critiques top out here unless the document supplies direct contradiction or a hard quantitative boundary.
|
|
49
|
+
- **100:** Use this exceptional anchor only when the document directly contradicts its own stated goal or violates its own hard quantitative boundary, and that contradiction will occur frequently if the plan is followed. It is the only anchor eligible for a silent fix; do not use it for qualitative complexity judgment.
|
|
48
50
|
|
|
49
51
|
## What you don't flag
|
|
50
52
|
|
|
@@ -26,9 +26,11 @@ Skip areas not relevant to the document's scope.
|
|
|
26
26
|
|
|
27
27
|
## Confidence calibration
|
|
28
28
|
|
|
29
|
-
- **
|
|
30
|
-
- **
|
|
31
|
-
- **
|
|
29
|
+
- **0:** The security concern is a false positive or a pre-existing issue. Suppress it.
|
|
30
|
+
- **25:** The threat might exist, but the plan and available context do not let you verify the exposure or missing control. Suppress it.
|
|
31
|
+
- **50:** The security gap is verified, but its impact is advisory or low and does not materially affect the described attack surface. Return it as FYI only.
|
|
32
|
+
- **75:** You have double-checked a concrete endpoint, trust boundary, input, secret, or data path against the plan and the missing control will directly affect security in practice. This is actionable.
|
|
33
|
+
- **100:** The plan directly confirms an unmitigated attack surface on a normal path, and the resulting exposure or exploit will occur frequently if implemented as written. Reserve this exceptional anchor for direct evidence and recurring impact; it is the only anchor eligible for a silent fix.
|
|
32
34
|
|
|
33
35
|
## What you don't flag
|
|
34
36
|
|
package/package.json
CHANGED
|
@@ -6,7 +6,7 @@ argument-hint: "[mode:headless] [path/to/document.md]"
|
|
|
6
6
|
|
|
7
7
|
# Document Review
|
|
8
8
|
|
|
9
|
-
Review requirements or plan documents through multi-persona analysis. Dispatches specialized reviewer agents in parallel,
|
|
9
|
+
Review requirements or plan documents through multi-persona analysis. Dispatches specialized reviewer agents in parallel, applies clear fixes, and presents proposed fixes and decisions for user input.
|
|
10
10
|
|
|
11
11
|
## Phase 0: Detect Mode
|
|
12
12
|
|
|
@@ -14,12 +14,13 @@ Check the skill arguments for `mode:headless`. Arguments may contain a document
|
|
|
14
14
|
|
|
15
15
|
If `mode:headless` is present, set **headless mode** for the rest of the workflow.
|
|
16
16
|
|
|
17
|
-
**Headless mode** changes the interaction model, not the classification boundaries. Document-review still applies the same judgment about what has one clear correct fix vs. what needs user judgment. The only difference is how
|
|
18
|
-
- `
|
|
19
|
-
- `
|
|
17
|
+
**Headless mode** changes the interaction model, not the classification boundaries. Document-review still applies the same judgment about what has one clear correct fix vs. what needs user judgment. The only difference is how findings are delivered:
|
|
18
|
+
- `safe_auto` fixes at confidence anchor `100` are applied silently (same as interactive)
|
|
19
|
+
- `gated_auto` proposed fixes and `manual` decisions are returned as structured text for the caller to handle -- no question prompts, no interactive approval
|
|
20
|
+
- Findings at confidence anchor `50` are returned as FYI observations only -- no interaction or action
|
|
20
21
|
- Phase 5 returns immediately with "Review complete" (no refine/complete question)
|
|
21
22
|
|
|
22
|
-
The caller receives
|
|
23
|
+
The caller receives applied fixes, proposed fixes, decisions, and FYI observations with their original classifications intact; proposed fixes and decisions remain for the caller to handle.
|
|
23
24
|
|
|
24
25
|
Callers invoke headless mode by including `mode:headless` in the skill arguments, e.g.:
|
|
25
26
|
```
|
|
@@ -133,7 +134,7 @@ Pass each agent the **full document** -- do not split into sections.
|
|
|
133
134
|
|
|
134
135
|
## Phases 3-5: Synthesis, Presentation, and Next Action
|
|
135
136
|
|
|
136
|
-
After all dispatched agents return, read `references/synthesis-and-presentation.md` for the synthesis pipeline (validate, gate, dedup, promote, resolve contradictions, route by autofix class),
|
|
137
|
+
After all dispatched agents return, read `references/synthesis-and-presentation.md` for the synthesis pipeline (validate, gate, dedup, promote, resolve contradictions, route by autofix class), fix application, finding presentation, and next-action menu. Do not load this file before agent dispatch completes.
|
|
137
138
|
|
|
138
139
|
---
|
|
139
140
|
|
|
@@ -45,8 +45,8 @@
|
|
|
45
45
|
},
|
|
46
46
|
"autofix_class": {
|
|
47
47
|
"type": "string",
|
|
48
|
-
"enum": ["
|
|
49
|
-
"description": "How this issue should be handled.
|
|
48
|
+
"enum": ["safe_auto", "gated_auto", "manual"],
|
|
49
|
+
"description": "How this issue should be handled. safe_auto = one mechanical, unambiguous fix suitable for silent application only at confidence 100; gated_auto = a codebase-pattern-resolved, factually incorrect, standard security or reliability, framework-native, or substantive mechanically implied fix that needs user confirmation; manual = multiple reasonable choices requiring user judgment."
|
|
50
50
|
},
|
|
51
51
|
"finding_type": {
|
|
52
52
|
"type": "string",
|
|
@@ -58,10 +58,9 @@
|
|
|
58
58
|
"description": "Concrete fix text. Omit or null if no good fix is obvious -- a bad suggestion is worse than none."
|
|
59
59
|
},
|
|
60
60
|
"confidence": {
|
|
61
|
-
"type": "
|
|
62
|
-
"
|
|
63
|
-
"
|
|
64
|
-
"maximum": 1.0
|
|
61
|
+
"type": "integer",
|
|
62
|
+
"enum": [0, 25, 50, 75, 100],
|
|
63
|
+
"description": "Reviewer confidence anchor: 0 = false positive or pre-existing issue; 25 = might be real but could not verify; 50 = verified real but nitpick, advisory, or not very important; 75 = double-checked, will hit in practice, and directly impacts correctness; 100 = evidence directly confirms and the issue will happen frequently."
|
|
65
64
|
},
|
|
66
65
|
"evidence": {
|
|
67
66
|
"type": "array",
|
|
@@ -12,56 +12,67 @@ Use this **exact format** when presenting synthesized review findings. Findings
|
|
|
12
12
|
**Document:** docs/plans/2026-03-15-feat-user-auth-plan.md
|
|
13
13
|
**Type:** plan
|
|
14
14
|
**Reviewers:** coherence, feasibility, security-lens, scope-guardian
|
|
15
|
-
- security-lens -- plan adds public API endpoint with auth flow
|
|
15
|
+
- security-lens -- plan adds a public API endpoint with an auth flow
|
|
16
16
|
- scope-guardian -- plan has 15 requirements across 3 priority levels
|
|
17
17
|
|
|
18
|
-
Applied
|
|
18
|
+
Applied 2 fixes. 4 items need attention (2 errors, 2 omissions). 1 FYI observation.
|
|
19
19
|
|
|
20
|
-
###
|
|
20
|
+
### Applied fixes
|
|
21
21
|
|
|
22
|
-
-
|
|
23
|
-
-
|
|
24
|
-
- Updated unit count from "6 units" to "7 units" to match listed units (coherence)
|
|
25
|
-
- Added "update API rate-limit config" step to Unit 4 -- implied by Unit 3's rate-limit introduction (feasibility)
|
|
26
|
-
- Added auth token refresh to test scenarios -- required by Unit 2's token expiry handling (security-lens)
|
|
22
|
+
- Corrected the stale cross-reference from Section 3.2 to Section 3.1 (coherence, confidence 100)
|
|
23
|
+
- Updated the unit count from 6 to 7 to match the listed units (feasibility, confidence 100)
|
|
27
24
|
|
|
28
25
|
### P0 -- Must Fix
|
|
29
26
|
|
|
30
27
|
#### Errors
|
|
31
28
|
|
|
29
|
+
**Decisions**
|
|
30
|
+
|
|
32
31
|
| # | Section | Issue | Reviewer | Confidence |
|
|
33
32
|
|---|---------|-------|----------|------------|
|
|
34
|
-
| 1 | Requirements Trace |
|
|
33
|
+
| 1 | Requirements Trace | The goal requires offline support, but the approach assumes persistent connectivity | coherence | 75 |
|
|
35
34
|
|
|
36
35
|
### P1 -- Should Fix
|
|
37
36
|
|
|
38
37
|
#### Errors
|
|
39
38
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
|
39
|
+
**Proposed fixes**
|
|
40
|
+
|
|
41
|
+
| # | Section | Issue | Suggested fix | Reviewer | Confidence |
|
|
42
|
+
|---|---------|-------|---------------|----------|------------|
|
|
43
|
+
| 2 | API Design | The plan claims Rails lacks native JSON error responses and proposes a custom serializer | Use Rails' existing `render json:` response path and remove the custom serializer step | feasibility | 75 |
|
|
43
44
|
|
|
44
45
|
#### Omissions
|
|
45
46
|
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
|
47
|
+
**Proposed fixes**
|
|
48
|
+
|
|
49
|
+
| # | Section | Issue | Suggested fix | Reviewer | Confidence |
|
|
50
|
+
|---|---------|-------|---------------|----------|------------|
|
|
51
|
+
| 3 | Implementation Unit 3 | The custom auth plan omits the existing Devise setup and migration path | Extend the existing Devise flow and document its migration path instead of introducing a parallel auth setup | feasibility | 100 |
|
|
49
52
|
|
|
50
53
|
### P2 -- Consider Fixing
|
|
51
54
|
|
|
52
55
|
#### Omissions
|
|
53
56
|
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
|
57
|
+
**Proposed fixes**
|
|
58
|
+
|
|
59
|
+
| # | Section | Issue | Suggested fix | Reviewer | Confidence |
|
|
60
|
+
|---|---------|-------|---------------|----------|------------|
|
|
61
|
+
| 4 | API Design | The public webhook endpoint has no rate limiting plan | Add the established request-throttling middleware to the webhook route and document its limit | security-lens | 75 |
|
|
62
|
+
|
|
63
|
+
### FYI observations
|
|
64
|
+
|
|
65
|
+
| # | Section | Observation | Reviewer | Confidence |
|
|
66
|
+
|---|---------|-------------|----------|------------|
|
|
67
|
+
| 5 | Error Handling | The plan does not say whether rate-limit responses include retry guidance | security-lens | 50 |
|
|
57
68
|
|
|
58
|
-
### Residual
|
|
69
|
+
### Residual concerns
|
|
59
70
|
|
|
60
71
|
| # | Concern | Source |
|
|
61
72
|
|---|---------|--------|
|
|
62
|
-
| 1 | Migration rollback strategy not addressed for Phase 2 data changes | feasibility |
|
|
73
|
+
| 1 | Migration rollback strategy is not addressed for Phase 2 data changes | feasibility |
|
|
63
74
|
|
|
64
|
-
### Deferred
|
|
75
|
+
### Deferred questions
|
|
65
76
|
|
|
66
77
|
| # | Question | Source |
|
|
67
78
|
|---|---------|--------|
|
|
@@ -69,21 +80,24 @@ Applied 5 auto-fixes. 4 findings to consider (2 errors, 2 omissions).
|
|
|
69
80
|
|
|
70
81
|
### Coverage
|
|
71
82
|
|
|
72
|
-
| Persona | Status | Findings |
|
|
73
|
-
|
|
74
|
-
| coherence | completed |
|
|
75
|
-
| feasibility | completed |
|
|
76
|
-
| security-lens | completed | 2 | 1 | 1 | 0 |
|
|
77
|
-
| scope-guardian | completed |
|
|
78
|
-
| product-lens | not activated | -- | -- | -- | -- |
|
|
79
|
-
| design-lens | not activated | -- | -- | -- | -- |
|
|
83
|
+
| Persona | Status | Findings | Fixes | Proposed fixes | Decisions | FYI observations | Residual |
|
|
84
|
+
|---------|--------|----------|-------|----------------|-----------|------------------|----------|
|
|
85
|
+
| coherence | completed | 2 | 1 | 0 | 1 | 0 | 0 |
|
|
86
|
+
| feasibility | completed | 3 | 1 | 2 | 0 | 0 | 1 |
|
|
87
|
+
| security-lens | completed | 2 | 0 | 1 | 0 | 1 | 0 |
|
|
88
|
+
| scope-guardian | completed | 0 | 0 | 0 | 0 | 0 | 1 |
|
|
89
|
+
| product-lens | not activated | -- | -- | -- | -- | -- | -- |
|
|
90
|
+
| design-lens | not activated | -- | -- | -- | -- | -- | -- |
|
|
80
91
|
```
|
|
81
92
|
|
|
82
93
|
## Section Rules
|
|
83
94
|
|
|
84
|
-
- **Summary line**: Always present after the reviewer list. Format: "Applied N
|
|
85
|
-
- **
|
|
95
|
+
- **Summary line**: Always present after the reviewer list. Format: "Applied N fixes. K items need attention (X errors, Y omissions). Z FYI observations." Omit any zero clause. `K` counts actionable proposed fixes and decisions; FYI observations are counted separately.
|
|
96
|
+
- **Applied fixes**: List all fixes applied silently (`safe_auto` at confidence `100`). Include enough detail per fix to convey the substance -- especially for fixes that add content or touch document meaning. A `safe_auto` finding at confidence `75` is demoted before routing and is not silently applied. Omit section if none.
|
|
86
97
|
- **P0-P3 sections**: Only include sections that have findings. Omit empty severity levels. Within each severity, separate into **Errors** and **Omissions** sub-headers. Omit a sub-header if that severity has none of that type.
|
|
87
|
-
- **
|
|
88
|
-
- **
|
|
89
|
-
- **
|
|
98
|
+
- **Proposed fixes**: Findings with `gated_auto` at confidence `75` or `100`. Include the concrete suggested fix and require user confirmation. Omit if none.
|
|
99
|
+
- **Decisions**: Findings with `manual` at confidence `75` or `100`. Include the suggested fix when one exists; otherwise present the judgment call without inventing a fix. Omit if none.
|
|
100
|
+
- **FYI observations**: Findings at confidence `50`, regardless of autofix class. They require no decision or action. Omit if none.
|
|
101
|
+
- **Residual concerns**: Unresolved residual risks that remain after restatement suppression, plus any residual items explicitly retained by synthesis. Omit if none.
|
|
102
|
+
- **Deferred questions**: Questions for later workflow stages. Omit if none.
|
|
103
|
+
- **Coverage**: Always include. All finding and route counts are **post-synthesis**. For each persona, **Findings** equals **Fixes + Proposed fixes + Decisions + FYI observations**. If deduplication merges a finding across personas, attribute it to the persona with the highest confidence anchor; if anchors tie, use document order, and reduce the other persona's finding and route counts. **Residual** remains the count of `residual_risks` from that persona's raw output, not the promoted or suppressed subset shown in Residual concerns. Failed or malformed reviewers are marked in Status and do not contribute findings.
|
|
@@ -26,18 +26,18 @@ Rules:
|
|
|
26
26
|
- Set `finding_type` for every finding:
|
|
27
27
|
- `error`: Something the document says that is wrong -- contradictions, incorrect statements, design tensions, incoherent tradeoffs.
|
|
28
28
|
- `omission`: Something the document forgot to say -- missing mechanical steps, absent list entries, undefined thresholds, forgotten cross-references.
|
|
29
|
+
- Set `confidence` to exactly one anchor from the schema, based on the evidence available:
|
|
30
|
+
- `0`: False positive or pre-existing issue. Suppress the finding.
|
|
31
|
+
- `25`: Might be real but could not verify. Suppress the finding.
|
|
32
|
+
- `50`: Verified real but nitpick, advisory, or not very important. This becomes an FYI observation.
|
|
33
|
+
- `75`: Double-checked, will hit in practice, and directly impacts correctness. This is actionable.
|
|
34
|
+
- `100`: Evidence directly confirms the issue and it will happen frequently. This is actionable and is the only anchor eligible for silent fixes.
|
|
29
35
|
- Set `autofix_class` based on whether there is one clear correct fix, not on severity or importance:
|
|
30
|
-
- `
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
- Missing standard security measures -- HTTPS enforcement, checksum verification, input sanitization, private IP rejection, or other controls with known implementations where omission is clearly a bug
|
|
36
|
-
- Incomplete technical descriptions -- the accurate/complete version is directly derivable from the codebase
|
|
37
|
-
- Missing requirements that follow mechanically from the document's own explicit, concrete decisions (not high-level goals -- a goal can be satisfied by multiple valid requirements)
|
|
38
|
-
The test is not "is this fix important?" but "is there more than one reasonable way to fix this?" If a competent implementer would arrive at the same fix independently, it is auto -- even if the fix is substantive. Always include `suggested_fix`. NOT auto if more than one reasonable fix exists or if scope/priority judgment is involved.
|
|
39
|
-
- `present`: Requires user judgment -- genuinely multiple valid approaches where the right choice depends on priorities, tradeoffs, or context the reviewer does not have. Examples: architectural choices with real tradeoffs, scope decisions, feature prioritization, UX design choices.
|
|
40
|
-
- `suggested_fix` is required for `auto` findings. For `present` findings, include only when the fix is obvious.
|
|
36
|
+
- `safe_auto`: A truly mechanical one-correct-fix case suitable for silent application only at anchor `100`: summary/detail mismatches, wrong counts, stale internal references, terminology drift, or additions mechanically implied by explicit content. Do not use this for codebase-pattern, factual, security/reliability, framework-native, or substantive completeness cases.
|
|
37
|
+
- `gated_auto`: A concrete fix resolved by an existing codebase pattern, factually incorrect behavior, a missing standard security or reliability control, a framework-native substitution, or a substantive mechanically implied completeness addition. The user confirms before applying it.
|
|
38
|
+
- `manual`: Multiple reasonable choices require user judgment, such as architectural tradeoffs, scope or priority decisions, feature prioritization, or UX choices.
|
|
39
|
+
The test is not "is this fix important?" but "is there more than one reasonable way to fix this?" If a competent implementer would arrive at the same fix independently, use `safe_auto` only for the truly mechanical cases above; use `gated_auto` when codebase or factual evidence resolves the choice but the fix is substantive. Do not classify a judgment call as automatic.
|
|
40
|
+
- `suggested_fix` is required for `safe_auto` and `gated_auto` findings. For `manual` findings, include it only when the fix is obvious.
|
|
41
41
|
- If you find no issues, return an empty findings array. Still populate residual_risks and deferred_questions if applicable.
|
|
42
42
|
- Use your suppress conditions. Do not flag issues that belong to other personas.
|
|
43
43
|
</output-contract>
|
|
@@ -9,14 +9,14 @@ Process findings from all agents through this pipeline. Order matters — each s
|
|
|
9
9
|
Check each agent's returned JSON against the findings schema:
|
|
10
10
|
|
|
11
11
|
- Drop findings missing any required field defined in the schema
|
|
12
|
-
- Drop findings with invalid enum values
|
|
12
|
+
- Drop findings with invalid enum values. Legacy `auto` / `present` values from older or in-flight dispatches are malformed; reject them without remapping rather than treating them as an alternate contract.
|
|
13
13
|
- Note the agent name for any malformed output in the Coverage section
|
|
14
14
|
|
|
15
15
|
**Do not narrate remap / validation diagnostics to the user.** Schema-drift notes ("persona X returned unknown enum Y, remapped to Z"), persona-prompt-drift commentary, and other validator-internal diagnostics are maintainer-facing information. They do not belong in the Phase 4 output the user reads. If a persona's output is malformed, the only user-visible consequence is a Coverage-row annotation (e.g., the persona shows fewer findings or a `malformed` marker). Everything else stays internal.
|
|
16
16
|
|
|
17
17
|
### 3.2 Confidence Gate (Anchor-Based)
|
|
18
18
|
|
|
19
|
-
Gate findings by their `confidence` anchor value. Anchors are discrete integers (`0`, `25`, `50`, `75`, `100`) with behavioral definitions documented in `references/findings-schema.json` and embedded in the persona rubric (`references/subagent-template.md`).
|
|
19
|
+
Gate findings by their `confidence` anchor value. Anchors are discrete integers (`0`, `25`, `50`, `75`, `100`) with behavioral definitions documented in `references/findings-schema.json` and embedded in the persona rubric (`references/subagent-template.md`). The anchors provide per-severity gates — doc-review economics do not warrant threshold gradation by severity, and coarse anchors prevent false-precision gaming. Continuous values from an older contract are malformed, not remapped.
|
|
20
20
|
|
|
21
21
|
| Anchor | Meaning | Route |
|
|
22
22
|
|--------|---------|-------|
|
|
@@ -174,7 +174,7 @@ Do NOT reclassify, re-route, or change the confidence anchor of any finding in t
|
|
|
174
174
|
- P1 manual "Migration lacks rollback strategy" — migration needs rollback regardless of scope. NOT linked (independence safeguard).
|
|
175
175
|
- P0 gated_auto "Deployment-ordering between migration and code" — concrete fix user confirms regardless. NOT linked (safeguard: gated_auto with own resolution path).
|
|
176
176
|
|
|
177
|
-
Result: 1 root + 4 dependents. User sees the root first; rejecting it cascades the 4 dependents
|
|
177
|
+
Result: 1 root + 4 dependents. User sees the root first; rejecting it cascades the 4 dependents out of the active decision set. Manual engagement drops from 11 → 7 (6 unlinked + 1 visible root).
|
|
178
178
|
|
|
179
179
|
**Worked example B (auth-shape).** Review of a plan to introduce a new session-management middleware. One finding is P1 manual "Middleware rewrite premise unsupported — existing session handling has no reported reliability issues" in Problem Frame. Scanning the other findings:
|
|
180
180
|
|