@dtlabs/ai-toolkit 0.1.4 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/agents/assessment-manager.md +274 -76
- package/.claude/agents/project-manager.md +2 -2
- package/.claude/skills/assess-codebase/SKILL.md +129 -4
- package/CHANGELOG.md +2 -0
- package/CLAUDE.md +1 -1
- package/docs/installation.md +1 -11
- package/docs/procedures/approval-gates.md +22 -14
- package/docs/procedures/assessment-findings-gate.md +120 -0
- package/docs/procedures/token-estimation.md +37 -5
- package/package.json +1 -1
- package/docs/procedures/assessment-approval-gate.md +0 -60
|
@@ -1,16 +1,17 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: assessment-manager
|
|
3
|
-
description: "Assessment Manager — intelligent orchestrator for the codebase assessment pipeline. Discovers available assessment agents, runs them in parallel, consolidates findings, gates on human approval,
|
|
3
|
+
description: "Assessment Manager — intelligent orchestrator for the codebase assessment pipeline. Discovers available assessment agents, runs them in parallel, consolidates findings into intervention documents, gates on human approval, and reports effort/token estimates. Input: path to target codebase (or '.' for current directory) [--scope=<area1,area2>] [--force]"
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Assessment Manager
|
|
7
7
|
|
|
8
|
-
An intelligent orchestrator that **assesses before it plans, and plans before it executes**. Discovers available assessment agents, runs them against the target codebase, consolidates findings, gates on human approval,
|
|
8
|
+
An intelligent orchestrator that **assesses before it plans, and plans before it executes**. Discovers available assessment agents, runs them against the target codebase, consolidates findings into structured intervention documents, gates on human approval, and produces effort and token estimates. Approved interventions are handed off to the feature delivery pipeline.
|
|
9
9
|
|
|
10
10
|
Before starting, read these procedures from `docs/procedures/`:
|
|
11
11
|
- `process-log.md` — log format and token tracking rules
|
|
12
12
|
- `issues-register.md` — Issues Register format and severity rules
|
|
13
|
-
- `assessment-
|
|
13
|
+
- `assessment-findings-gate.md` — gate presentation, approvals file format, rules
|
|
14
|
+
- `token-estimation.md` — Token-Estimate.md format, estimation model, and assessment pipeline input size estimates
|
|
14
15
|
|
|
15
16
|
Also read `docs/pricing.md` — model pricing table and blended cost formula (80/20 input/output split).
|
|
16
17
|
|
|
@@ -26,7 +27,7 @@ Scan `.claude/agents/` and read each file's `description` frontmatter. Classify
|
|
|
26
27
|
- **Documentation** — produces structured documents (keywords: "documentation standard", "intervention")
|
|
27
28
|
- **Orchestrator** — skip self-invocation
|
|
28
29
|
|
|
29
|
-
Build
|
|
30
|
+
Build an `assessment_agents` registry.
|
|
30
31
|
|
|
31
32
|
### 1b. Determine assessment prefix
|
|
32
33
|
|
|
@@ -59,15 +60,13 @@ Target codebase
|
|
|
59
60
|
└── intervention-documentation-standard
|
|
60
61
|
│
|
|
61
62
|
├── {PREFIX}-INT-NNN-*.md (one per finding)
|
|
62
|
-
|
|
63
|
+
├── {PREFIX}-Interventions-Index.md
|
|
64
|
+
├── {PREFIX}-Effort-Estimate.md (finalised)
|
|
65
|
+
└── {PREFIX}-Token-Estimate.md (updated)
|
|
63
66
|
│
|
|
64
67
|
══ REMEDIATION GATE ══ → {PREFIX}-Approvals.md
|
|
65
68
|
│
|
|
66
|
-
|
|
67
|
-
│
|
|
68
|
-
review-solution per intervention → commit on pass
|
|
69
|
-
│
|
|
70
|
-
Pull Request (feature/{PREFIX}-remediation → develop)
|
|
69
|
+
Pipeline ends → Summary
|
|
71
70
|
```
|
|
72
71
|
|
|
73
72
|
### Planning rules
|
|
@@ -75,8 +74,6 @@ Target codebase
|
|
|
75
74
|
1. All assessment agents run in **parallel** — read-only, no mutual dependencies
|
|
76
75
|
2. `intervention-documentation-standard` runs **after** all assessments complete
|
|
77
76
|
3. Skip agents with fresh output unless `--force`
|
|
78
|
-
4. Remediation agents dispatch **only after** the Remediation Gate is approved
|
|
79
|
-
5. Parallelize remediation agents where interventions have no dependency on each other
|
|
80
77
|
|
|
81
78
|
### Show plan to user before executing
|
|
82
79
|
|
|
@@ -89,9 +86,8 @@ Target codebase
|
|
|
89
86
|
concurrency-safety-assessment → ASSESS-001-Concurrency-Assessment.md
|
|
90
87
|
|
|
91
88
|
⏳ QUEUE: intervention-documentation-standard → ASSESS-001-INT-*.md + Interventions-Index.md
|
|
92
|
-
⏳
|
|
93
|
-
⏳
|
|
94
|
-
⏳ PR: feature/ASSESS-001-remediation → develop
|
|
89
|
+
⏳ WRITE: ASSESS-001-Effort-Estimate.md + ASSESS-001-Token-Estimate.md
|
|
90
|
+
⏳ GATE: Review findings → acknowledge + flag interventions
|
|
95
91
|
─────────────────────────────────────────────────────
|
|
96
92
|
```
|
|
97
93
|
|
|
@@ -105,6 +101,178 @@ After all complete:
|
|
|
105
101
|
- Verify each expected output file exists and is non-empty
|
|
106
102
|
- If an agent produced no output, log the failure and continue — partial assessments are better than none
|
|
107
103
|
|
|
104
|
+
### Write initial Token Estimate file
|
|
105
|
+
|
|
106
|
+
After verifying outputs and before moving to Phase 4, accumulate token usage from every completed assessment agent and write `{PREFIX}-Token-Estimate.md` to `docs/assessments/{PREFIX}/`. Create the directory if it does not exist.
|
|
107
|
+
|
|
108
|
+
**Step 1 — Check estimation model (mandatory).** Confirm `docs/procedures/token-estimation.md` is present and readable. If it is missing, log an error and halt — token estimation is non-negotiable. Parse the file to extract the canonical parameters:
|
|
109
|
+
|
|
110
|
+
| Parameter | Value |
|
|
111
|
+
|-----------|-------|
|
|
112
|
+
| Avg chars per token | 4 |
|
|
113
|
+
| Haiku system prompt | ~2,000 tokens |
|
|
114
|
+
| Sonnet system prompt | ~3,000 tokens |
|
|
115
|
+
| Base overhead per call | ~5,000 tokens |
|
|
116
|
+
|
|
117
|
+
Store these parameters in memory and reuse them for all estimation calculations in Phases 3–4. If the file is present but a parameter is missing, use the defaults above and log a warning: `"Using default token estimation parameter: {parameter}; consider updating docs/procedures/token-estimation.md"`.
|
|
118
|
+
|
|
119
|
+
**Step 2 — Check pricing data (graceful).** Confirm `docs/pricing.md` is present and readable. If it is missing or malformed, log a warning: `"docs/pricing.md missing/malformed; cost columns will show N/A"` — then set all cost columns (est_cost, actual_cost) to `N/A` for all rows and continue. Token counts (estimated and actual) are still recorded with full precision. Phase subtotals and grand total cost rows show `N/A`. The pipeline does not halt. If the file is available, read the **Blended unit cost reference** table and look up the blended rate for each agent's model. Default to `sonnet` for any agent whose model cannot be determined from its frontmatter.
|
|
120
|
+
|
|
121
|
+
**Step 3 — Estimate tokens per assessment agent.** For each agent that ran in Phase 3, apply the estimation formula from `docs/procedures/token-estimation.md` (Assessment pipeline usage section):
|
|
122
|
+
|
|
123
|
+
```
|
|
124
|
+
est_tokens = base_overhead + system_prompt_weight + (input_size_chars / 4)
|
|
125
|
+
= 5,000 + 3,000 (sonnet) + (input_size_chars / 4)
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
Use the agent-specific input size estimates from `docs/procedures/token-estimation.md`. The blended rate for all assessment agents is `$0.005400/1k tokens` (sonnet). Compute:
|
|
129
|
+
|
|
130
|
+
```
|
|
131
|
+
est_cost = est_tokens × (blended_rate / 1000)
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Round `est_tokens` to the nearest integer; format `est_cost` to 4 decimal places (e.g., `$0.0444`).
|
|
135
|
+
|
|
136
|
+
**Step 4 — Extract actual tokens per assessment agent.** For each completed assessment agent, read its `<usage>` block from its result:
|
|
137
|
+
|
|
138
|
+
- If the block is present: `actual_tokens = input_tokens + output_tokens`. Compute `actual_cost = actual_tokens × (blended_rate / 1000)` at 4 decimal places.
|
|
139
|
+
- If the block is missing or unparseable: set `actual_tokens = "N/A"` and `actual_cost = "N/A"`. Log a warning: `"[agent_name] produced no <usage> block; token data unavailable"`. Exclude this agent from phase subtotal and grand total calculations.
|
|
140
|
+
|
|
141
|
+
Status remains `"complete"` in both cases — a missing usage block does not indicate agent failure.
|
|
142
|
+
|
|
143
|
+
**Step 5 — Compute Phase 3 subtotals.** Sum `est_tokens` and `est_cost` across all assessment agents (all rows, regardless of N/A status). Sum `actual_tokens` and `actual_cost` only across rows where actual values are numeric (exclude N/A rows). Format subtotal costs to 2 decimal places.
|
|
144
|
+
|
|
145
|
+
**Step 6 — Write `{PREFIX}-Token-Estimate.md`.** Write the complete file at `docs/assessments/{PREFIX}/{PREFIX}-Token-Estimate.md` using the template below. Replace all `{PREFIX}`, `{N}`, and `{placeholder}` markers with computed values.
|
|
146
|
+
|
|
147
|
+
```markdown
|
|
148
|
+
# Token Estimate — {PREFIX} — Assessment Pipeline
|
|
149
|
+
|
|
150
|
+
> Estimates computed before execution. Actuals accumulated as agents complete.
|
|
151
|
+
> Assessment agents (Phase 3) actuals: filled at end of phase.
|
|
152
|
+
> Intervention documentation (Phase 4) actuals: filled on completion.
|
|
153
|
+
> Orchestrator row added by assess-codebase skill at pipeline end.
|
|
154
|
+
> Pricing model: docs/pricing.md (80% input / 20% output split).
|
|
155
|
+
|
|
156
|
+
## Estimation model
|
|
157
|
+
|
|
158
|
+
| Parameter | Value |
|
|
159
|
+
|-----------|-------|
|
|
160
|
+
| Avg chars per token | 4 |
|
|
161
|
+
| Haiku system prompt | ~2,000 tokens |
|
|
162
|
+
| Sonnet system prompt | ~3,000 tokens |
|
|
163
|
+
| Base overhead per call | ~5,000 tokens |
|
|
164
|
+
| Input/output split | 80% / 20% |
|
|
165
|
+
|
|
166
|
+
## Assessment agents (Phase 3)
|
|
167
|
+
|
|
168
|
+
| Agent | Model | Est. tokens | Est. cost ($) | Actual tokens | Actual cost ($) | Status |
|
|
169
|
+
|-------|-------|-------------|---------------|---------------|-----------------|--------|
|
|
170
|
+
{one row per assessment agent that ran — computed values from Steps 3 and 4 — status = "complete"}
|
|
171
|
+
|
|
172
|
+
## Intervention documentation (Phase 4)
|
|
173
|
+
|
|
174
|
+
> To be populated after Phase 4 completion.
|
|
175
|
+
|
|
176
|
+
| Agent | Model | Est. tokens | Est. cost ($) | Actual tokens | Actual cost ($) | Status |
|
|
177
|
+
|-------|-------|-------------|---------------|---------------|-----------------|--------|
|
|
178
|
+
|
|
179
|
+
## Remediation
|
|
180
|
+
|
|
181
|
+
Remediation effort is tracked separately via the feature delivery pipeline (not part of this assessment).
|
|
182
|
+
|
|
183
|
+
## Phase subtotals
|
|
184
|
+
|
|
185
|
+
| Phase | Est. tokens | Est. cost ($) | Actual tokens | Actual cost ($) |
|
|
186
|
+
|-------|-------------|---------------|---------------|-----------------|
|
|
187
|
+
| Assessment (Phase 3) | {est_tokens_total} | ${est_cost_total} | {actual_tokens_total} | ${actual_cost_total} |
|
|
188
|
+
| Intervention documentation (Phase 4) | — | — | — | — |
|
|
189
|
+
|
|
190
|
+
## Grand total
|
|
191
|
+
|
|
192
|
+
> Partial — updated at pipeline end.
|
|
193
|
+
|
|
194
|
+
| Metric | Estimated | Actual |
|
|
195
|
+
|--------|-----------|--------|
|
|
196
|
+
| Total tokens | {est_tokens_total} | partial — updated at pipeline end |
|
|
197
|
+
| Total cost ($) | ${est_cost_total_2dp} | partial — updated at pipeline end |
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
After writing, verify the file is readable: attempt to read it back. If it is unreadable, log an error: `"[timestamp] ERROR: Token Estimate file unreadable after write — {path}: {error}"`. If the write itself fails (e.g., permission denied, disk full), log the error: `"[timestamp] ERROR: Failed to write Token Estimate file — {path}: {error}"` and continue — the pipeline does not halt due to a Token Estimate file I/O failure.
|
|
201
|
+
|
|
202
|
+
**Step 7 — Log the file write.** Append to the process log:
|
|
203
|
+
|
|
204
|
+
```
|
|
205
|
+
[timestamp] Phase 3 end: wrote Token Estimate file
|
|
206
|
+
Assessment agents: {N} rows ({completed} complete, {na_count} with N/A actuals)
|
|
207
|
+
Location: docs/assessments/{PREFIX}/{PREFIX}-Token-Estimate.md
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Where `{N}` is the total number of assessment agents that ran, `{completed}` is the count with numeric actual tokens, and `{na_count}` is the count with `N/A` actual tokens.
|
|
211
|
+
|
|
212
|
+
### Write initial Effort Estimate file
|
|
213
|
+
|
|
214
|
+
**Step 1 — Read process log timestamps.** For each completed Phase 3 assessment agent, scan `{PREFIX}-process-log.txt` for lines matching:
|
|
215
|
+
- `Agent START: {agent_name}` → extract start timestamp
|
|
216
|
+
- `Agent DONE: {agent_name}` → extract end timestamp
|
|
217
|
+
|
|
218
|
+
If both timestamps are present, compute `actual_duration = end_timestamp − start_timestamp`, rounded to the nearest minute. Express as `"Xmin"` or `"Xh Ymin"` (e.g., `"12min"`, `"1h 5min"`). If either timestamp is missing, set `actual_duration = "N/A"` and log a warning: `"[agent_name] has no timestamp in the process log; actual duration unavailable"`.
|
|
219
|
+
|
|
220
|
+
For the batch wall-clock:
|
|
221
|
+
- `batch_start = minimum of all Phase 3 agent start timestamps`
|
|
222
|
+
- `batch_end = maximum of all Phase 3 agent end timestamps`
|
|
223
|
+
- `batch_actual = batch_end − batch_start`, rounded to nearest minute
|
|
224
|
+
|
|
225
|
+
**Step 2 — Compute estimates.** On the first run (no prior `{PREFIX}-Effort-Estimate.md` exists), set `est_duration = "N/A"` for all agent rows. On subsequent runs, extract the `actual_duration` from the previous run's rows for each agent and use as the new `est_duration`. If the prior file has `"N/A"` for an agent, continue using `"N/A"` as the estimate.
|
|
226
|
+
|
|
227
|
+
**Step 3 — Compute delta.** `delta = actual_duration − est_duration`. If either is `"N/A"`, set `delta = "N/A"`. For numeric values, express as `"-Xmin"` (faster), `"+Xmin"` (slower), or `"0min"` (on target).
|
|
228
|
+
|
|
229
|
+
**Step 4 — Write `{PREFIX}-Effort-Estimate.md`.** Write to `docs/assessments/{PREFIX}/{PREFIX}-Effort-Estimate.md`. Create the directory if it does not exist.
|
|
230
|
+
|
|
231
|
+
```markdown
|
|
232
|
+
# Effort Estimate — {PREFIX} — Assessment Pipeline
|
|
233
|
+
|
|
234
|
+
> Wall-clock effort tracking for the assessment pipeline.
|
|
235
|
+
> Assessment agent durations: filled at end of Phase 3.
|
|
236
|
+
> Intervention documentation duration: filled at end of Phase 4.
|
|
237
|
+
> Remediation effort: estimated from Interventions Index at end of Phase 4 using fixed rates.
|
|
238
|
+
> Actual remediation effort tracked by feature delivery pipeline per intervention.
|
|
239
|
+
> Effort rates: CRITICAL=8h, HIGH=4h, MEDIUM=2h, LOW=1h (human hours, sequential).
|
|
240
|
+
|
|
241
|
+
## Assessment phase
|
|
242
|
+
|
|
243
|
+
| Agent | Est. duration | Actual duration | Delta | Status |
|
|
244
|
+
|-------|--------------|-----------------|-------|--------|
|
|
245
|
+
| {agent_name} | {est_duration} | {actual_duration} | {delta} | {status} |
|
|
246
|
+
... (one row per Phase 3 agent — intervention-documentation-standard appended at Phase 4 end)
|
|
247
|
+
|
|
248
|
+
## Assessment phase subtotal
|
|
249
|
+
|
|
250
|
+
| Metric | Estimated | Actual | Delta |
|
|
251
|
+
|--------|-----------|--------|-------|
|
|
252
|
+
| Phase 3 assessment batch | {est_batch} | {actual_batch} | {delta_batch} |
|
|
253
|
+
| intervention-documentation-standard | {est} | {actual} | {delta} |
|
|
254
|
+
| Total | {total_est} | {total_actual} | {total_delta} |
|
|
255
|
+
|
|
256
|
+
> Note: Phase 3 "Actual" row uses batch wall-clock (max end − min start), not sum of individual durations.
|
|
257
|
+
> Individual agent durations are in the assessment phase table above.
|
|
258
|
+
|
|
259
|
+
## Remediation effort estimate
|
|
260
|
+
|
|
261
|
+
> Pending intervention documentation completion.
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
If a scope filter was applied, append to the header block: `> Note: Scope filter applied ({scope}). This estimate reflects only {scope} assessment areas.`
|
|
265
|
+
|
|
266
|
+
After writing, verify the file is readable. If unreadable, log an error: `"[timestamp] ERROR: Effort Estimate file unreadable after write — {path}: {error}"`. If the write fails, log: `"[timestamp] ERROR: Failed to write Effort Estimate file — {path}: {error}"` and continue — the pipeline does not halt due to a file I/O failure.
|
|
267
|
+
|
|
268
|
+
**Step 5 — Log the write.** Append to the process log:
|
|
269
|
+
|
|
270
|
+
```
|
|
271
|
+
[timestamp] Phase 3 end: wrote Effort Estimate file
|
|
272
|
+
Assessment agents: {N} rows ({completed} with actuals, {na_count} with N/A)
|
|
273
|
+
Location: docs/assessments/{PREFIX}/{PREFIX}-Effort-Estimate.md
|
|
274
|
+
```
|
|
275
|
+
|
|
108
276
|
---
|
|
109
277
|
|
|
110
278
|
## Phase 4 — Intervention Document Generation
|
|
@@ -113,90 +281,122 @@ Invoke `intervention-documentation-standard` with all produced assessment files
|
|
|
113
281
|
- `{PREFIX}-INT-NNN-{slug}.md` per intervention
|
|
114
282
|
- `{PREFIX}-Interventions-Index.md` (ordered by criticality and dependency)
|
|
115
283
|
|
|
116
|
-
|
|
284
|
+
### Append intervention-documentation row to Token Estimate file
|
|
117
285
|
|
|
118
|
-
|
|
286
|
+
After `intervention-documentation-standard` completes, update the Token Estimate file as follows.
|
|
119
287
|
|
|
120
|
-
|
|
288
|
+
**Step 1 — Extract actual tokens.** Read the `<usage>` block from the agent result: `actual_tokens = input_tokens + output_tokens`. Compute `actual_cost = actual_tokens × (blended_rate / 1000)` at 4 decimal places. If the block is missing or unparseable: set `actual_tokens = "N/A"` and `actual_cost = "N/A"`, and log a warning: `"intervention-documentation-standard produced no <usage> block; token data unavailable"`. Status remains `"complete"` regardless.
|
|
121
289
|
|
|
122
|
-
|
|
290
|
+
**Step 2 — Estimate tokens.** Apply the standard formula:
|
|
291
|
+
|
|
292
|
+
```
|
|
293
|
+
est_tokens = base_overhead (5,000) + system_prompt_weight (3,000 for sonnet) + (intervention_doc_size_bytes / 4)
|
|
294
|
+
```
|
|
123
295
|
|
|
124
|
-
|
|
296
|
+
If the total size of produced intervention documents is unknown at estimation time, use `~15,500` tokens as a typical estimate. Compute `est_cost = est_tokens × (blended_rate / 1000)` at 4 decimal places. If pricing data is unavailable, set both cost fields to `N/A`.
|
|
297
|
+
|
|
298
|
+
**Step 3 — Replace the placeholder section.** Open `{PREFIX}-Token-Estimate.md`. Find the "Intervention documentation (Phase 4)" section (the `> To be populated after Phase 4 completion.` placeholder row). Replace the empty table body with the actual row:
|
|
299
|
+
|
|
300
|
+
| Agent | Model | Est. tokens | Est. cost ($) | Actual tokens | Actual cost ($) | Status |
|
|
301
|
+
|-------|-------|-------------|---------------|---------------|-----------------|--------|
|
|
302
|
+
| intervention-documentation-standard | sonnet | {est_tokens} | {est_cost} | {actual_tokens} | {actual_cost} | complete |
|
|
303
|
+
|
|
304
|
+
Remove the `> To be populated after Phase 4 completion.` note once the row is written.
|
|
305
|
+
|
|
306
|
+
**Step 4 — Log the completion.** Append to the process log:
|
|
307
|
+
|
|
308
|
+
```
|
|
309
|
+
[timestamp] Phase 4 end: appended intervention-documentation-standard row to Token Estimate file. Actual tokens: {N} | Cost: ${X.XXXX}
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
### Update Effort Estimate file
|
|
313
|
+
|
|
314
|
+
**Step 1 — Read timestamps and compute duration.** Scan `{PREFIX}-process-log.txt` for `Agent START: intervention-documentation-standard` and `Agent DONE: intervention-documentation-standard`. Compute `actual_duration` (same formula as Phase 3 agents). If missing, set `actual_duration = "N/A"` and log a warning.
|
|
315
|
+
|
|
316
|
+
**Step 2 — Append intervention-documentation-standard row.** Open `{PREFIX}-Effort-Estimate.md` and append one row to the "Assessment phase" table:
|
|
125
317
|
|
|
126
|
-
### 6a. Create remediation branch
|
|
127
318
|
```
|
|
128
|
-
|
|
319
|
+
| intervention-documentation-standard | {est_duration} | {actual_duration} | {delta} | complete |
|
|
129
320
|
```
|
|
130
|
-
Branch from `develop`.
|
|
131
321
|
|
|
132
|
-
|
|
322
|
+
`est_duration` follows the same first-run / subsequent-run logic as Phase 3 agents.
|
|
323
|
+
|
|
324
|
+
**Step 3 — Update assessment phase subtotal.** Fill in the `intervention-documentation-standard` row in the subtotal table and compute the Total row values.
|
|
325
|
+
|
|
326
|
+
**Step 4 — Build remediation section.** Read `{PREFIX}-Interventions-Index.md` to extract intervention counts by severity. Apply fixed rates: CRITICAL × 8h, HIGH × 4h, MEDIUM × 2h, LOW × 1h. Sum for the human sequential total. If the Interventions Index is missing, write `"Interventions Index not found — remediation estimate unavailable"` as the remediation section body, log a warning, and continue. If all counts are 0, add: `> Note: No remediation required — zero findings identified.`
|
|
133
327
|
|
|
134
|
-
|
|
328
|
+
**Step 5 — Replace placeholder.** Replace the `> Pending intervention documentation completion.` line in the "Remediation effort estimate" section with:
|
|
135
329
|
|
|
136
|
-
|
|
330
|
+
```markdown
|
|
331
|
+
## Remediation effort estimate
|
|
137
332
|
|
|
138
|
-
|
|
333
|
+
> Derived from {PREFIX}-Interventions-Index.md. Rates: CRITICAL=8h, HIGH=4h, MEDIUM=2h, LOW=1h (human hours, sequential).
|
|
334
|
+
> Actual remediation effort tracked by feature delivery pipeline per intervention.
|
|
139
335
|
|
|
140
|
-
|
|
336
|
+
| Severity | Count | Rate | Subtotal |
|
|
337
|
+
|----------|-------|------|---------|
|
|
338
|
+
| CRITICAL | {count} | 8h | {subtotal} |
|
|
339
|
+
| HIGH | {count} | 4h | {subtotal} |
|
|
340
|
+
| MEDIUM | {count} | 2h | {subtotal} |
|
|
341
|
+
| LOW | {count} | 1h | {subtotal} |
|
|
342
|
+
| **Total** | **{total_count}** | — | **{total_hours}h** |
|
|
141
343
|
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
344
|
+
Human sequential total: {total_hours}h
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
**Step 6 — Validate severity count.** Assert `sum(CRITICAL + HIGH + MEDIUM + LOW)` equals total intervention count in `{PREFIX}-Interventions-Index.md`. If they differ, log: `"Severity count mismatch between Effort Estimate and Interventions Index — check {PREFIX}-Interventions-Index.md"`. Use the Interventions Index counts as the authoritative source.
|
|
348
|
+
|
|
349
|
+
**Step 7 — Log the update.** Append to the process log:
|
|
350
|
+
|
|
351
|
+
```
|
|
352
|
+
[timestamp] Phase 4 end: updated Effort Estimate with intervention-documentation-standard row
|
|
353
|
+
Actual duration: {X}min
|
|
354
|
+
[timestamp] Phase 4 end: populated remediation effort section
|
|
355
|
+
CRITICAL: {N}, HIGH: {N}, MEDIUM: {N}, LOW: {N}
|
|
356
|
+
Human sequential total: {X}h
|
|
357
|
+
Location: docs/assessments/{PREFIX}/{PREFIX}-Effort-Estimate.md
|
|
358
|
+
```
|
|
149
359
|
|
|
150
360
|
---
|
|
151
361
|
|
|
152
|
-
## Phase
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
## Remediation
|
|
164
|
-
- Interventions approved: N / N proposed
|
|
165
|
-
- Interventions implemented: N
|
|
166
|
-
- Issues Register: {N} fixed, {N} deferred → see {PREFIX}-Issues.md
|
|
167
|
-
|
|
168
|
-
## Test plan
|
|
169
|
-
- [ ] Build passes
|
|
170
|
-
- [ ] Test suite passes
|
|
171
|
-
- [ ] Manual verification of critical fixes
|
|
172
|
-
EOF
|
|
173
|
-
)"
|
|
174
|
-
```
|
|
362
|
+
## Phase 5 — Findings Gate
|
|
363
|
+
|
|
364
|
+
Follow `docs/procedures/assessment-findings-gate.md`. The gate has two mandatory steps:
|
|
365
|
+
|
|
366
|
+
**Step 5a — Acknowledge:** present findings summary (including effort estimate) and wait for acknowledgement. Accept any non-empty text reply.
|
|
367
|
+
|
|
368
|
+
**Step 5b — Flag interventions:** prompt for INT-NNN identifiers to flag for feature delivery. Validate each against `{PREFIX}-Interventions-Index.md`; re-prompt if any are unknown. Accept "None" for zero flagged.
|
|
369
|
+
|
|
370
|
+
After both steps complete, write `{PREFIX}-Approvals.md` per the format in `docs/procedures/assessment-findings-gate.md`. Verify the file was written before proceeding to Phase 6.
|
|
371
|
+
|
|
372
|
+
No writes to the Effort Estimate file during Phase 5.
|
|
175
373
|
|
|
176
374
|
---
|
|
177
375
|
|
|
178
|
-
## Phase
|
|
376
|
+
## Phase 6 — Summary
|
|
377
|
+
|
|
378
|
+
Report:
|
|
179
379
|
|
|
180
380
|
```
|
|
181
|
-
|
|
381
|
+
Assessment Manager — Run Summary
|
|
182
382
|
─────────────────────────────────────────────────────
|
|
183
383
|
Target: {codebase path} | Prefix: {PREFIX}
|
|
184
384
|
─────────────────────────────────────────────────────
|
|
185
385
|
Assessment:
|
|
186
|
-
✅
|
|
187
|
-
✅
|
|
188
|
-
|
|
386
|
+
✅ {agent_1} → {PREFIX}-{output}.md
|
|
387
|
+
✅ {agent_2} → {PREFIX}-{output}.md
|
|
388
|
+
...
|
|
189
389
|
─────────────────────────────────────────────────────
|
|
190
390
|
Findings: N CRITICAL | N HIGH | N MEDIUM | N LOW
|
|
191
|
-
Interventions: N proposed | N
|
|
391
|
+
Interventions: N proposed | N flagged for feature delivery
|
|
192
392
|
─────────────────────────────────────────────────────
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
393
|
+
Approvals: {PREFIX}-Approvals.md
|
|
394
|
+
Effort Estimate: {PREFIX}-Effort-Estimate.md
|
|
395
|
+
Token Estimate: {PREFIX}-Token-Estimate.md
|
|
396
|
+
Process log: docs/assessments/{PREFIX}/{PREFIX}-process-log.txt
|
|
196
397
|
─────────────────────────────────────────────────────
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
Process log: docs/assessments/{PREFIX}/{PREFIX}-process-log.txt
|
|
398
|
+
Flagged interventions can be actioned via /define-feature
|
|
399
|
+
referencing the INT-NNN document.
|
|
200
400
|
─────────────────────────────────────────────────────
|
|
201
401
|
```
|
|
202
402
|
|
|
@@ -204,15 +404,13 @@ Process log: docs/assessments/{PREFIX}/{PREFIX}-process-log.txt
|
|
|
204
404
|
|
|
205
405
|
## Guidelines
|
|
206
406
|
|
|
207
|
-
- **Read all procedures at startup** — `process-log.md`, `issues-register.md`, `assessment-
|
|
208
|
-
- **
|
|
407
|
+
- **Read all procedures at startup** — `process-log.md`, `issues-register.md`, `assessment-findings-gate.md`, `token-estimation.md`
|
|
408
|
+
- **Token tracking — maintain an internal ledger keyed by agent name.** For each completed agent call: extract the `<usage>` block (input_tokens, output_tokens, model), compute `actual_tokens = input_tokens + output_tokens`, and record the result. If the block is absent or unparseable, record `actual_tokens = "N/A"` and log a warning — never halt because of a missing usage block. Estimate tokens for each agent BEFORE dispatch using `est_tokens = base_overhead (5,000) + system_prompt_weight (3,000 for sonnet, 2,000 for haiku) + (input_size_chars / 4)` and compute blended cost using the formula from `docs/pricing.md`. Exclude N/A rows from all cost totals and accuracy statistics; include them in the file as visible rows.
|
|
209
409
|
- **Assessment agents are read-only** — never modify source files during assessment phases
|
|
210
|
-
- **Never skip the
|
|
410
|
+
- **Never skip the Findings Gate** — it is mandatory regardless of finding count; `--force` re-presents it, never bypasses it
|
|
411
|
+
- **Assessment pipeline is read-only** — no branch creation, no code changes, no commits, no PRs
|
|
211
412
|
- **Assessment agents run in parallel** — they have no mutual dependencies
|
|
212
|
-
- **Prefer specialised remediation agents** over generic developer agents when available
|
|
213
413
|
- **Plan before executing** — always show the plan first
|
|
214
414
|
- **Fail fast on missing target** — if the target path does not exist, stop immediately
|
|
215
415
|
- **Partial assessments are acceptable** — if one agent fails, continue with others
|
|
216
|
-
- **Commit per intervention** — immediately after it passes review
|
|
217
|
-
- **PR targets `develop`** — never push directly; do NOT auto-merge
|
|
218
416
|
- **All output documents in English**
|
|
@@ -52,7 +52,7 @@ state = {
|
|
|
52
52
|
|
|
53
53
|
### 2a. Cross-feature analysis
|
|
54
54
|
|
|
55
|
-
Compare `feature.md` against each `registry` entry for OVERLAP, DEPENDENCY, or CONFLICT (see `docs/procedures/feature-registry.md`). Show the cross-feature report to the user. If conflicts are found, use `AskUserQuestion`
|
|
55
|
+
Compare `feature.md` against each `registry` entry for OVERLAP, DEPENDENCY, or CONFLICT (see `docs/procedures/feature-registry.md`). Show the cross-feature report to the user. If conflicts are found, output a hard-stop message and wait for a text reply before continuing. Do NOT use `AskUserQuestion` — it cannot be answered through the agent relay chain. If `registry` is empty, output "No prior features — no cross-reference needed."
|
|
56
56
|
|
|
57
57
|
### 2b. Build execution plan
|
|
58
58
|
|
|
@@ -301,7 +301,7 @@ To add a new agent: create `.claude/agents/my-agent.md` with a clear `descriptio
|
|
|
301
301
|
- **Plan before executing** — always show the plan first
|
|
302
302
|
- **Never re-run fresh agents** unless `--force`
|
|
303
303
|
- **Parallelize where possible** — independent agents run concurrently
|
|
304
|
-
- **Gates are hard stops** —
|
|
304
|
+
- **Gates are hard stops** — output the hard-stop text and wait for a text reply from the user; write `{PREFIX}-Approvals.md`, verify it on disk. Do NOT use `AskUserQuestion` — it cannot be answered through the agent relay chain. Never auto-approve, never assume, never skip
|
|
305
305
|
- **Pre-condition check at Phase 6** — read `{PREFIX}-Approvals.md` and verify both gates ✅ before touching the codebase. If missing, return to the gate
|
|
306
306
|
- **Review at US level** — not per task
|
|
307
307
|
- **Commit per US** — `feat({PREFIX}): implement US-XX — {title}`
|
|
@@ -1,10 +1,18 @@
|
|
|
1
1
|
---
|
|
2
|
-
description: "Assess Codebase — starts the
|
|
2
|
+
description: "Assess Codebase — starts the codebase assessment pipeline (parallel assessment → findings consolidation → intervention documents → findings gate). Usage: /assess-codebase [path] [--scope=architecture,security,quality,concurrency,devops] [--force]"
|
|
3
3
|
argument-hint: "[path] [--scope=architecture,security,quality,concurrency,devops] [--force]"
|
|
4
|
-
disable-model-invocation: true
|
|
5
4
|
---
|
|
6
5
|
|
|
7
|
-
|
|
6
|
+
# Assess Codebase
|
|
7
|
+
|
|
8
|
+
Orchestrates the full codebase assessment pipeline by spawning the `assessment-manager` agent
|
|
9
|
+
and, once it completes, recording the real token consumption in a dedicated file.
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## Step 1 — Spawn the Assessment Manager
|
|
14
|
+
|
|
15
|
+
Spawn the `assessment-manager` agent with the exact arguments the user provided:
|
|
8
16
|
|
|
9
17
|
```
|
|
10
18
|
subagent_type: assessment-manager
|
|
@@ -13,4 +21,121 @@ prompt: <path> [--scope=...] [--force]
|
|
|
13
21
|
|
|
14
22
|
If no path is provided, use `.` (current working directory).
|
|
15
23
|
|
|
16
|
-
|
|
24
|
+
Wait for the agent to complete. Capture its full result, including the `<usage>` block
|
|
25
|
+
that appears at the end of the tool result (format: `subagent_tokens: N`).
|
|
26
|
+
|
|
27
|
+
The assessment-manager handles the full pipeline internally — it presents the Phase 5 Findings Gate
|
|
28
|
+
to the user, records the acknowledgement and flagged interventions, and ends at the summary.
|
|
29
|
+
The skill waits for the agent to complete and proceeds to Step 2 regardless of outcome.
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## Step 2 — Extract orchestrator token usage
|
|
34
|
+
|
|
35
|
+
From the `<usage>` block of the assessment-manager result, read:
|
|
36
|
+
- `subagent_tokens` — total tokens consumed by the assessment-manager agent itself (input + output)
|
|
37
|
+
- `duration_ms` — wall-clock duration of the assessment-manager run
|
|
38
|
+
|
|
39
|
+
If the `<usage>` block is missing: log warning
|
|
40
|
+
`"assessment-manager produced no <usage> block; orchestrator token data unavailable"`
|
|
41
|
+
and set `actual_tokens = "N/A"`.
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## Step 3 — Complete Token Estimate file
|
|
46
|
+
|
|
47
|
+
The assessment-manager will have written `{PREFIX}-Token-Estimate.md` at
|
|
48
|
+
`docs/assessments/{PREFIX}/{PREFIX}-Token-Estimate.md` with assessment and
|
|
49
|
+
intervention-documentation rows. Read that file, then **append** the following in order:
|
|
50
|
+
|
|
51
|
+
### 3a — Orchestrator row
|
|
52
|
+
|
|
53
|
+
The orchestrator row is NOT added to any of the three phase-level tables (Assessment agents, Intervention documentation, Remediation agents). It appears only in the Actuals vs Estimate section (Step 3b below).
|
|
54
|
+
|
|
55
|
+
For use in Step 3b, compute the orchestrator row values as follows:
|
|
56
|
+
|
|
57
|
+
- Estimated tokens: 80,000 (baseline from estimation model)
|
|
58
|
+
- Estimated cost: $0.4320 — computed as `80,000 × $0.005400 / 1,000` using the sonnet blended
|
|
59
|
+
rate from `docs/pricing.md`. If `docs/pricing.md` is missing, use `"N/A"` for est_cost.
|
|
60
|
+
- Actual tokens: value from `subagent_tokens` in the `<usage>` block; `"N/A"` if block was missing
|
|
61
|
+
- Actual cost: `(actual_tokens / 1,000) × blended_rate_per_1k`, formatted to 4 decimal places; `"N/A"` if actual_tokens is `"N/A"` or pricing data is unavailable
|
|
62
|
+
|
|
63
|
+
### 3b — Actuals vs Estimate section
|
|
64
|
+
|
|
65
|
+
Append a horizontal rule (`---`) followed by an Actuals vs Estimate section. Include one
|
|
66
|
+
row for every agent that ran: all assessment agents, intervention-documentation-standard,
|
|
67
|
+
and the assessment-manager orchestrator.
|
|
68
|
+
|
|
69
|
+
```markdown
|
|
70
|
+
## Actuals vs Estimate
|
|
71
|
+
|
|
72
|
+
| Agent | Task/Scope | Model | Est. tokens | Actual tokens | Delta | Est. cost ($) | Actual cost ($) | Duration |
|
|
73
|
+
|-------|------------|-------|-------------|---------------|-------|---------------|-----------------|----------|
|
|
74
|
+
| {agent} | {scope} | {model} | {est} | {actual} | {±delta} | {est_cost} | {actual_cost} | {Xmin Ys} |
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Column rules:
|
|
78
|
+
- **Delta**: `actual_tokens − est_tokens`; show as `+N` or `−N`. Show `"N/A"` if either
|
|
79
|
+
value is missing.
|
|
80
|
+
- **Duration**: convert `duration_ms` to `Xmin Ys` where available; use `"—"` where not.
|
|
81
|
+
- Rows with `"N/A"` actual tokens are still shown in the table but excluded from any
|
|
82
|
+
aggregate calculations.
|
|
83
|
+
|
|
84
|
+
### 3c — Estimation accuracy by agent type (conditional)
|
|
85
|
+
|
|
86
|
+
If 2 or more distinct model tiers appear in the rows that have non-`"N/A"` actual tokens
|
|
87
|
+
(e.g., haiku AND sonnet), append:
|
|
88
|
+
|
|
89
|
+
```markdown
|
|
90
|
+
## Estimation accuracy by agent type
|
|
91
|
+
|
|
92
|
+
| Model | Count | Avg est. tokens | Avg actual tokens | Avg delta | Trend |
|
|
93
|
+
|-------|-------|-----------------|-------------------|-----------|-------|
|
|
94
|
+
| {model} | {N} | {avg_est} | {avg_actual} | {avg_delta} | {trend} |
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Column rules:
|
|
98
|
+
- One row per model tier (e.g., `haiku`, `sonnet`)
|
|
99
|
+
- Exclude rows with `"N/A"` actual tokens from all averages
|
|
100
|
+
- **Trend**: `"over-target"` if avg delta > +20% of avg est; `"under-target"` if avg delta
|
|
101
|
+
< −20% of avg est; otherwise `"on-target"`
|
|
102
|
+
|
|
103
|
+
### 3d — Grand Total section
|
|
104
|
+
|
|
105
|
+
Update the Grand Total section already written in the file: replace the
|
|
106
|
+
`"partial — updated at pipeline end"` marker with `"Final"` and fill in the computed values:
|
|
107
|
+
|
|
108
|
+
```markdown
|
|
109
|
+
## Grand Total (Final)
|
|
110
|
+
|
|
111
|
+
| Metric | Estimated | Actual | Delta | Delta % |
|
|
112
|
+
|--------|-----------|--------|-------|---------|
|
|
113
|
+
| Total tokens (all agents) | {sum_est} | {sum_actual} | ±{delta} | {delta_pct} |
|
|
114
|
+
| Total cost ($) | ${sum_est_cost} | ${sum_actual_cost} | ±${delta_cost} | {delta_cost_pct} |
|
|
115
|
+
| Total wall-clock | — | {Xmin Ys} | — | — |
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Column rules:
|
|
119
|
+
- **Delta %**: `(delta / estimated) × 100` at 1 decimal place; show `"N/A"` if estimated is 0
|
|
120
|
+
|
|
121
|
+
Summation rules:
|
|
122
|
+
- Sum all agents' tokens and costs; exclude `"N/A"` rows from sums
|
|
123
|
+
- Cost values: 2 decimal places for subtotals and totals; 4 decimal places for per-row costs
|
|
124
|
+
- The "Remediation" section in the Token Estimate file contains a static note (not a placeholder); leave it as-is
|
|
125
|
+
- Wall-clock: convert `duration_ms` from the assessment-manager result to minutes/seconds
|
|
126
|
+
|
|
127
|
+
---
|
|
128
|
+
|
|
129
|
+
## Step 4 — Report to user
|
|
130
|
+
|
|
131
|
+
After writing to the Token Estimate file, report:
|
|
132
|
+
|
|
133
|
+
```
|
|
134
|
+
Assessment pipeline complete.
|
|
135
|
+
Token estimate + actuals → {PREFIX}-Token-Estimate.md
|
|
136
|
+
Approvals → docs/assessments/{PREFIX}/{PREFIX}-Approvals.md
|
|
137
|
+
Process log → docs/assessments/{PREFIX}/{PREFIX}-process-log.txt
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
If the Token Estimate file does not exist (assessment-manager failed before writing it):
|
|
141
|
+
note this in the report but do not halt — the skill has completed its work.
|
package/CHANGELOG.md
CHANGED
package/CLAUDE.md
CHANGED
|
@@ -114,7 +114,7 @@ Generic procedures live in `docs/procedures/` in this toolkit. Projects can over
|
|
|
114
114
|
|
|
115
115
|
| Agent | subagent_type | Purpose |
|
|
116
116
|
|-------|---------------|---------|
|
|
117
|
-
| Assessment Manager | `assessment-manager` | Orchestrates the
|
|
117
|
+
| Assessment Manager | `assessment-manager` | Orchestrates the assessment pipeline: parallel assessment → intervention docs → approval gate → effort/token estimates → summary |
|
|
118
118
|
| Generic Software Assessment | `generic-software-assessment` | Broad quality analysis across architecture, security, testability, observability, DevOps |
|
|
119
119
|
| Intervention Documentation Standard | `intervention-documentation-standard` | Generates structured, self-contained intervention documents from assessment findings |
|
|
120
120
|
| God Class Decomposition | `god-class-decomposition` | Safe incremental decomposition of oversized classes and methods |
|
package/docs/installation.md
CHANGED
|
@@ -33,7 +33,7 @@ npx @dtlabs/ai-toolkit --local . --force
|
|
|
33
33
|
|
|
34
34
|
### Global installation
|
|
35
35
|
|
|
36
|
-
Merges `.claude/agents`, `.claude/skills`, `docs/`, and `CLAUDE.md` into `~/.claude/`:
|
|
36
|
+
Merges `.claude/agents`, `.claude/skills`, `.claude/commands`, `docs/`, and `CLAUDE.md` into `~/.claude/`:
|
|
37
37
|
|
|
38
38
|
```bash
|
|
39
39
|
npm install -g @dtlabs/ai-toolkit
|
|
@@ -68,13 +68,3 @@ ai-toolkit --global --force
|
|
|
68
68
|
```bash
|
|
69
69
|
npm view @dtlabs/ai-toolkit version
|
|
70
70
|
```
|
|
71
|
-
|
|
72
|
-
---
|
|
73
|
-
|
|
74
|
-
## Publishing a new version
|
|
75
|
-
|
|
76
|
-
```bash
|
|
77
|
-
npm login # authenticate with your npmjs.com account
|
|
78
|
-
npm version patch # bump version (patch / minor / major)
|
|
79
|
-
npm publish --access public # --access public is required for scoped packages
|
|
80
|
-
```
|
|
@@ -16,15 +16,19 @@ Show the user a summary of the generated documents:
|
|
|
16
16
|
|
|
17
17
|
### Step 2 — Request explicit sign-off
|
|
18
18
|
|
|
19
|
-
|
|
19
|
+
Output the following hard-stop message and **wait for a text reply from the user**. Do NOT use `AskUserQuestion` — it cannot be answered through the agent relay chain.
|
|
20
20
|
|
|
21
|
-
|
|
22
|
-
|
|
21
|
+
```
|
|
22
|
+
⛔ GATE 1 — DOCS APPROVAL — HARD STOP
|
|
23
|
+
|
|
24
|
+
Please review the documents above and reply with one of:
|
|
23
25
|
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
26
|
+
"Approve — proceed to Work Breakdown"
|
|
27
|
+
"Request changes: <describe what to change>"
|
|
28
|
+
"Approve with notes: <your comments>"
|
|
29
|
+
|
|
30
|
+
The pipeline CANNOT continue until you reply directly.
|
|
31
|
+
```
|
|
28
32
|
|
|
29
33
|
### Step 3 — Write approval record BEFORE continuing
|
|
30
34
|
|
|
@@ -81,15 +85,19 @@ Show key metrics:
|
|
|
81
85
|
|
|
82
86
|
### Step 3 — Request explicit sign-off
|
|
83
87
|
|
|
84
|
-
|
|
88
|
+
Output the following hard-stop message and **wait for a text reply from the user**. Do NOT use `AskUserQuestion` — it cannot be answered through the agent relay chain.
|
|
85
89
|
|
|
86
|
-
|
|
87
|
-
|
|
90
|
+
```
|
|
91
|
+
⛔ GATE 2 — WORK BREAKDOWN APPROVAL — HARD STOP
|
|
92
|
+
|
|
93
|
+
Please review the Work Breakdown above and reply with one of:
|
|
88
94
|
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
95
|
+
"Approve — start implementation"
|
|
96
|
+
"Request changes: <describe what to change>"
|
|
97
|
+
"Approve with notes: <your comments>"
|
|
98
|
+
|
|
99
|
+
Implementation CANNOT start until you reply directly.
|
|
100
|
+
```
|
|
93
101
|
|
|
94
102
|
### Step 4 — Update approval record BEFORE starting implementation
|
|
95
103
|
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
# Procedure: Assessment Findings Gate
|
|
2
|
+
|
|
3
|
+
After intervention documents are generated, present findings to the user and obtain explicit acknowledgement before closing the pipeline. The pipeline is read-only — no remediation code is written.
|
|
4
|
+
|
|
5
|
+
## Gate presentation
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
📊 Assessment Complete — {PREFIX}
|
|
9
|
+
═══════════════════════════════════════
|
|
10
|
+
Assessment results:
|
|
11
|
+
Generic quality: [KPI summary scores]
|
|
12
|
+
Layer violations: N CRITICAL, N HIGH, N MEDIUM
|
|
13
|
+
Concurrency hazards: N CRITICAL, N HIGH
|
|
14
|
+
[other assessments]
|
|
15
|
+
|
|
16
|
+
Interventions proposed: N total
|
|
17
|
+
CRITICAL: N (immediate attention)
|
|
18
|
+
HIGH: N (next sprint)
|
|
19
|
+
MEDIUM: N (planned cycle)
|
|
20
|
+
LOW: N (backlog)
|
|
21
|
+
|
|
22
|
+
Interventions Index: docs/assessments/{PREFIX}/{PREFIX}-Interventions-Index.md
|
|
23
|
+
|
|
24
|
+
Remediation Effort Estimate (from {PREFIX}-Effort-Estimate.md)
|
|
25
|
+
─────────────────────────────────────────────────────────────
|
|
26
|
+
CRITICAL: N × 8h = Xh
|
|
27
|
+
HIGH: N × 4h = Xh
|
|
28
|
+
MEDIUM: N × 2h = Xh
|
|
29
|
+
LOW: N × 1h = Xh
|
|
30
|
+
Total estimated (human sequential): Xh
|
|
31
|
+
─────────────────────────────────────────────────────────────
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Step 5a — Mandatory acknowledgement
|
|
35
|
+
|
|
36
|
+
Output the following hard-stop message and **wait for a text reply from the user**. Do NOT use `AskUserQuestion` — it cannot be answered through the agent relay chain.
|
|
37
|
+
|
|
38
|
+
```
|
|
39
|
+
⛔ FINDINGS GATE — STEP 1 OF 2 — HARD STOP
|
|
40
|
+
|
|
41
|
+
Please review the Interventions Index above.
|
|
42
|
+
|
|
43
|
+
Reply with "Acknowledged" (or any text) to confirm you have reviewed the findings.
|
|
44
|
+
|
|
45
|
+
The pipeline CANNOT continue until you reply directly.
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Accept any non-empty text reply as acknowledgement. Record the timestamp.
|
|
49
|
+
|
|
50
|
+
## Step 5b — Optional intervention flagging
|
|
51
|
+
|
|
52
|
+
Immediately after acknowledgement, output:
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
FINDINGS GATE — STEP 2 OF 2
|
|
56
|
+
|
|
57
|
+
Which interventions (if any) do you want to flag for feature delivery?
|
|
58
|
+
|
|
59
|
+
Reply with one of:
|
|
60
|
+
"None" — no interventions flagged
|
|
61
|
+
"INT-001, INT-003, ..." — comma-separated list of INT-NNN identifiers
|
|
62
|
+
|
|
63
|
+
Flagged interventions will be recorded in {PREFIX}-Approvals.md.
|
|
64
|
+
Any intervention can be actioned later via /define-feature referencing its INT-NNN document.
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
**Validation:** For each INT-NNN the user provides, verify it exists in `{PREFIX}-Interventions-Index.md`. If any identifier is unknown, report it and re-prompt Step 5b without proceeding:
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
Unknown identifier(s): INT-099
|
|
71
|
+
Please re-enter — only identifiers from {PREFIX}-Interventions-Index.md are valid.
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Re-prompt until all listed identifiers are valid, or the user replies "None".
|
|
75
|
+
|
|
76
|
+
## Approvals file format
|
|
77
|
+
|
|
78
|
+
Write `{PREFIX}-Approvals.md` in the assessment directory **only after both steps complete successfully**. Never write a partial file.
|
|
79
|
+
|
|
80
|
+
```markdown
|
|
81
|
+
# Findings Acknowledgement — {PREFIX}
|
|
82
|
+
|
|
83
|
+
## Assessment Reviewed
|
|
84
|
+
|
|
85
|
+
| Document | Status | Date | Notes |
|
|
86
|
+
|----------|--------|------|-------|
|
|
87
|
+
| {PREFIX}-Generic-Assessment.md | Acknowledged | {date} | — |
|
|
88
|
+
| {PREFIX}-Interventions-Index.md | Acknowledged | {date} | — |
|
|
89
|
+
|
|
90
|
+
## Interventions Flagged for Feature Delivery
|
|
91
|
+
|
|
92
|
+
| Intervention | Flagged | Date | Notes |
|
|
93
|
+
|---|---|---|---|
|
|
94
|
+
| INT-001 — {slug} | Yes | {date} | — |
|
|
95
|
+
| INT-002 — {slug} | No | {date} | Not selected |
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
Rules:
|
|
99
|
+
- Every intervention from `{PREFIX}-Interventions-Index.md` must appear — one row per intervention
|
|
100
|
+
- "Flagged: Yes" for each INT-NNN the user explicitly named in Step 5b
|
|
101
|
+
- "Flagged: No" for all others, with Notes = "Not selected"
|
|
102
|
+
- If zero interventions were flagged, add a note below the table: `> No interventions selected for feature delivery.`
|
|
103
|
+
- If zero interventions exist (empty index), write the table header only and add: `> No interventions proposed — zero findings identified.`
|
|
104
|
+
|
|
105
|
+
## Process log entries
|
|
106
|
+
|
|
107
|
+
```
|
|
108
|
+
[timestamp] Phase 5 gate: presented findings (N interventions)
|
|
109
|
+
[timestamp] Phase 5 gate: Step 5a — user acknowledged findings
|
|
110
|
+
[timestamp] Phase 5 gate: Step 5b — user flagged: INT-001, INT-003 (or "none")
|
|
111
|
+
[timestamp] Phase 5 gate: wrote {PREFIX}-Approvals.md
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
## Rules
|
|
115
|
+
|
|
116
|
+
- **Never skip this gate** — it is mandatory regardless of finding count
|
|
117
|
+
- **Never write a partial Approvals file** — both steps must complete before writing
|
|
118
|
+
- **`--force` does not bypass the gate** — it re-presents the gate and overwrites the existing Approvals file
|
|
119
|
+
- **Log every gate event** in the process log (gate opened, step 5a complete, step 5b result, file written)
|
|
120
|
+
- **No remediation** — this gate never triggers branch creation, agent dispatch, commits, or PRs
|
|
@@ -10,6 +10,7 @@ Write `{PREFIX}-Token-Estimate.md` in the feature directory after Work Breakdown
|
|
|
10
10
|
| Agent system prompt weight — haiku | ~2,000 tokens |
|
|
11
11
|
| Agent system prompt weight — sonnet | ~3,000 tokens |
|
|
12
12
|
| Base overhead per agent call | ~5,000 tokens |
|
|
13
|
+
| Input/output split | 80% / 20% |
|
|
13
14
|
|
|
14
15
|
## Cost estimation
|
|
15
16
|
|
|
@@ -98,10 +99,10 @@ The 80/20 input/output split is defined in `docs/pricing.md` — update there if
|
|
|
98
99
|
|
|
99
100
|
## Estimation accuracy by agent type
|
|
100
101
|
|
|
101
|
-
|
|
|
102
|
-
|
|
103
|
-
| haiku
|
|
104
|
-
| sonnet
|
|
102
|
+
| Model | Count | Avg est. tokens | Avg actual tokens | Avg delta | Trend |
|
|
103
|
+
|-------|-------|----------------|------------------|-----------|-------|
|
|
104
|
+
| haiku | N | N | N | ±N | ↑/↓/→ |
|
|
105
|
+
| sonnet | N | N | N | ±N | ↑/↓/→ |
|
|
105
106
|
|
|
106
107
|
## Grand Total
|
|
107
108
|
|
|
@@ -117,7 +118,38 @@ The 80/20 input/output split is defined in `docs/pricing.md` — update there if
|
|
|
117
118
|
- One row per invocation — rework invocations get a "(rework)" suffix
|
|
118
119
|
- Delta (tokens): negative = fewer tokens than estimated, positive = more
|
|
119
120
|
- Cost: computed with blended formula from `docs/pricing.md`; use 4 decimal places for per-agent rows, 2 decimal places for totals
|
|
120
|
-
- Trend:
|
|
121
|
+
- Trend: "↑" if avg delta > +5% of avg est, "↓" if avg delta < -5% of avg est, otherwise "→"
|
|
121
122
|
- Duration: convert `duration_ms` to minutes/seconds
|
|
122
123
|
- Model: read from agent frontmatter, default to `sonnet` if not set
|
|
123
124
|
- Missing `<usage>` block → write `N/A`, exclude from accuracy stats and cost totals
|
|
125
|
+
|
|
126
|
+
## Assessment pipeline usage
|
|
127
|
+
|
|
128
|
+
The `assessment-manager` agent reads this procedure at Phase 3 start before dispatching assessment agents, in the same way that `project-manager` reads it before dispatching implementation agents. If this file is missing, the assessment pipeline halts with an error — this is the only non-graceful failure mode in the pipeline.
|
|
129
|
+
|
|
130
|
+
### Assessment agent input size estimates
|
|
131
|
+
|
|
132
|
+
All assessment agents and the `intervention-documentation-standard` agent run on **sonnet** — use the `$0.005400/1k tokens` blended rate from `docs/pricing.md`.
|
|
133
|
+
|
|
134
|
+
Apply the standard formula with the code input sizes below:
|
|
135
|
+
|
|
136
|
+
```
|
|
137
|
+
est_tokens = base_overhead + system_prompt_weight + (input_size_chars / 4)
|
|
138
|
+
= 5,000 + 3,000 (sonnet) + (input_size_chars / 4)
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
| Agent | Approx. input size | Input tokens (chars ÷ 4) | Est. total tokens |
|
|
142
|
+
|-------|--------------------|--------------------------|-------------------|
|
|
143
|
+
| generic-software-assessment | ~80,000 chars of code | ~20,000 | ~28,000 |
|
|
144
|
+
| layered-architecture-assessment | ~50,000 chars of code | ~12,500 | ~20,500 |
|
|
145
|
+
| concurrency-safety-assessment | ~40,000 chars of code | ~10,000 | ~18,000 |
|
|
146
|
+
| other assessment agents | ~30,000 chars of code | ~7,500 | ~15,500 |
|
|
147
|
+
| intervention-documentation-standard | ~30,000 chars of assessment findings | ~7,500 | ~15,500 |
|
|
148
|
+
|
|
149
|
+
### Remediation agents
|
|
150
|
+
|
|
151
|
+
Remediation agents receive an intervention document (~20,000 chars, ~5,000 tokens for the doc portion) plus relevant code sections. Use the intervention doc portion as the base input estimate and add code section tokens using the standard formula. Typical total range: 13,000–25,000 tokens depending on intervention scope.
|
|
152
|
+
|
|
153
|
+
### Orchestrator (assessment-manager)
|
|
154
|
+
|
|
155
|
+
Estimate as the sum of all child agent result sizes (as input) plus process log writes (as output), following the same orchestrator rule as `project-manager`. Use **80,000 tokens** as a conservative baseline for a full pipeline run (assessment + intervention documentation + remediation phases combined).
|
package/package.json
CHANGED
|
@@ -1,60 +0,0 @@
|
|
|
1
|
-
# Procedure: Assessment Remediation Gate
|
|
2
|
-
|
|
3
|
-
After intervention documents are generated, present findings to the user and obtain explicit approval before any remediation code is written.
|
|
4
|
-
|
|
5
|
-
## Gate presentation
|
|
6
|
-
|
|
7
|
-
```
|
|
8
|
-
📊 Assessment Complete — ASSESS-001
|
|
9
|
-
═══════════════════════════════════════
|
|
10
|
-
Assessment results:
|
|
11
|
-
Generic quality: [KPI summary scores]
|
|
12
|
-
Layer violations: N CRITICAL, N HIGH, N MEDIUM
|
|
13
|
-
Concurrency hazards: N CRITICAL, N HIGH
|
|
14
|
-
[other assessments]
|
|
15
|
-
|
|
16
|
-
Interventions proposed: N total
|
|
17
|
-
CRITICAL: N (immediate attention)
|
|
18
|
-
HIGH: N (next sprint)
|
|
19
|
-
MEDIUM: N (planned cycle)
|
|
20
|
-
LOW: N (backlog)
|
|
21
|
-
|
|
22
|
-
Interventions Index: docs/assessments/ASSESS-001/ASSESS-001-Interventions-Index.md
|
|
23
|
-
```
|
|
24
|
-
|
|
25
|
-
Use `AskUserQuestion`:
|
|
26
|
-
- **"Approve all"** — all proposed interventions approved
|
|
27
|
-
- **"Approve CRITICAL only"** — only CRITICAL severity approved
|
|
28
|
-
- **"Select interventions"** — user specifies INT-NNN to approve (comma-separated)
|
|
29
|
-
- **"Assessment only"** — stop here, no remediation
|
|
30
|
-
|
|
31
|
-
If the user selects **"Assessment only"**, skip to the Summary phase.
|
|
32
|
-
|
|
33
|
-
## Approvals file format
|
|
34
|
-
|
|
35
|
-
Write `{PREFIX}-Approvals.md` in the assessment directory:
|
|
36
|
-
|
|
37
|
-
```markdown
|
|
38
|
-
# Approval Record — {PREFIX}
|
|
39
|
-
|
|
40
|
-
## Assessment Approvals
|
|
41
|
-
|
|
42
|
-
| Document | Status | Date | Notes |
|
|
43
|
-
|----------|--------|------|-------|
|
|
44
|
-
| {PREFIX}-Generic-Assessment.md | ✅ Reviewed | {date} | — |
|
|
45
|
-
| {PREFIX}-Interventions-Index.md | ✅ Reviewed | {date} | — |
|
|
46
|
-
|
|
47
|
-
## Remediation Scope
|
|
48
|
-
|
|
49
|
-
| Intervention | Approved | Date | Notes |
|
|
50
|
-
|---|---|---|---|
|
|
51
|
-
| INT-001 — sql-injection-hardening | ✅ Approved | {date} | — |
|
|
52
|
-
| INT-002 — god-class-decomposition | ✅ Approved | {date} | — |
|
|
53
|
-
| INT-003 — di-refactoring | ❌ Deferred | {date} | Out of current sprint scope |
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
## Rules
|
|
57
|
-
|
|
58
|
-
- **Never skip this gate** — remediation cannot start without explicit approval
|
|
59
|
-
- **Log the approval outcome** in the process log
|
|
60
|
-
- **Record every decision** including deferred interventions and the reason
|