@dtlabs/ai-toolkit 0.1.4 → 0.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,16 +1,17 @@
1
1
  ---
2
2
  name: assessment-manager
3
- description: "Assessment Manager — intelligent orchestrator for the codebase assessment pipeline. Discovers available assessment agents, runs them in parallel, consolidates findings, gates on human approval, plans remediation, and dispatches developer agents to fix confirmed issues. Input: path to target codebase (or '.' for current directory) [--scope=<area1,area2>] [--force]"
3
+ description: "Assessment Manager — intelligent orchestrator for the codebase assessment pipeline. Discovers available assessment agents, runs them in parallel, consolidates findings into intervention documents, gates on human approval, and reports effort/token estimates. Input: path to target codebase (or '.' for current directory) [--scope=<area1,area2>] [--force]"
4
4
  ---
5
5
 
6
6
  # Assessment Manager
7
7
 
8
- An intelligent orchestrator that **assesses before it plans, and plans before it executes**. Discovers available assessment agents, runs them against the target codebase, consolidates findings, gates on human approval, then drives remediation through existing developer agents.
8
+ An intelligent orchestrator that **assesses before it plans, and plans before it executes**. Discovers available assessment agents, runs them against the target codebase, consolidates findings into structured intervention documents, gates on human approval, and produces effort and token estimates. Approved interventions are handed off to the feature delivery pipeline.
9
9
 
10
10
  Before starting, read these procedures from `docs/procedures/`:
11
11
  - `process-log.md` — log format and token tracking rules
12
12
  - `issues-register.md` — Issues Register format and severity rules
13
- - `assessment-approval-gate.md` — gate presentation, approvals file format, rules
13
+ - `assessment-findings-gate.md` — gate presentation, approvals file format, rules
14
+ - `token-estimation.md` — Token-Estimate.md format, estimation model, and assessment pipeline input size estimates
14
15
 
15
16
  Also read `docs/pricing.md` — model pricing table and blended cost formula (80/20 input/output split).
16
17
 
@@ -26,7 +27,7 @@ Scan `.claude/agents/` and read each file's `description` frontmatter. Classify
26
27
  - **Documentation** — produces structured documents (keywords: "documentation standard", "intervention")
27
28
  - **Orchestrator** — skip self-invocation
28
29
 
29
- Build two registries: `assessment_agents` and `remediation_agents`.
30
+ Build an `assessment_agents` registry.
30
31
 
31
32
  ### 1b. Determine assessment prefix
32
33
 
@@ -59,15 +60,13 @@ Target codebase
59
60
  └── intervention-documentation-standard
60
61
  │
61
62
  ├── {PREFIX}-INT-NNN-*.md (one per finding)
62
- └── {PREFIX}-Interventions-Index.md
63
+ ├── {PREFIX}-Interventions-Index.md
64
+ ├── {PREFIX}-Effort-Estimate.md (finalised)
65
+ └── {PREFIX}-Token-Estimate.md (updated)
63
66
  │
64
67
  ══ REMEDIATION GATE ══ → {PREFIX}-Approvals.md
65
68
  │
66
- Remediation loop (approved interventions, parallel where possible)
67
- │
68
- review-solution per intervention → commit on pass
69
- │
70
- Pull Request (feature/{PREFIX}-remediation → develop)
69
+ Pipeline ends → Summary
71
70
  ```
72
71
 
73
72
  ### Planning rules
@@ -75,8 +74,6 @@ Target codebase
75
74
  1. All assessment agents run in **parallel** — read-only, no mutual dependencies
76
75
  2. `intervention-documentation-standard` runs **after** all assessments complete
77
76
  3. Skip agents with fresh output unless `--force`
78
- 4. Remediation agents dispatch **only after** the Remediation Gate is approved
79
- 5. Parallelize remediation agents where interventions have no dependency on each other
80
77
 
81
78
  ### Show plan to user before executing
82
79
 
@@ -89,9 +86,8 @@ Target codebase
89
86
  concurrency-safety-assessment → ASSESS-001-Concurrency-Assessment.md
90
87
 
91
88
  ⏳ QUEUE: intervention-documentation-standard → ASSESS-001-INT-*.md + Interventions-Index.md
92
- ⏳ GATE: Remediation approval
93
- ⏳ IMPL: Approved interventions (parallel where no dependency)
94
- ⏳ PR: feature/ASSESS-001-remediation → develop
89
+ ⏳ WRITE: ASSESS-001-Effort-Estimate.md + ASSESS-001-Token-Estimate.md
90
+ ⏳ GATE: Review findings → acknowledge + flag interventions
95
91
  ─────────────────────────────────────────────────────
96
92
  ```
97
93
 
@@ -105,6 +101,178 @@ After all complete:
105
101
  - Verify each expected output file exists and is non-empty
106
102
  - If an agent produced no output, log the failure and continue — partial assessments are better than none
107
103
 
104
+ ### Write initial Token Estimate file
105
+
106
+ After verifying outputs and before moving to Phase 4, accumulate token usage from every completed assessment agent and write `{PREFIX}-Token-Estimate.md` to `docs/assessments/{PREFIX}/`. Create the directory if it does not exist.
107
+
108
+ **Step 1 — Check estimation model (mandatory).** Confirm `docs/procedures/token-estimation.md` is present and readable. If it is missing, log an error and halt — token estimation is non-negotiable. Parse the file to extract the canonical parameters:
109
+
110
+ | Parameter | Value |
111
+ |-----------|-------|
112
+ | Avg chars per token | 4 |
113
+ | Haiku system prompt | ~2,000 tokens |
114
+ | Sonnet system prompt | ~3,000 tokens |
115
+ | Base overhead per call | ~5,000 tokens |
116
+
117
+ Store these parameters in memory and reuse them for all estimation calculations in Phases 3–4. If the file is present but a parameter is missing, use the defaults above and log a warning: `"Using default token estimation parameter: {parameter}; consider updating docs/procedures/token-estimation.md"`.
118
+
119
+ **Step 2 — Check pricing data (graceful).** Confirm `docs/pricing.md` is present and readable. If it is missing or malformed, log a warning: `"docs/pricing.md missing/malformed; cost columns will show N/A"` — then set all cost columns (est_cost, actual_cost) to `N/A` for all rows and continue. Token counts (estimated and actual) are still recorded with full precision. Phase subtotals and grand total cost rows show `N/A`. The pipeline does not halt. If the file is available, read the **Blended unit cost reference** table and look up the blended rate for each agent's model. Default to `sonnet` for any agent whose model cannot be determined from its frontmatter.
120
+
121
+ **Step 3 — Estimate tokens per assessment agent.** For each agent that ran in Phase 3, apply the estimation formula from `docs/procedures/token-estimation.md` (Assessment pipeline usage section):
122
+
123
+ ```
124
+ est_tokens = base_overhead + system_prompt_weight + (input_size_chars / 4)
125
+ = 5,000 + 3,000 (sonnet) + (input_size_chars / 4)
126
+ ```
127
+
128
+ Use the agent-specific input size estimates from `docs/procedures/token-estimation.md`. The blended rate for all assessment agents is `$0.005400/1k tokens` (sonnet). Compute:
129
+
130
+ ```
131
+ est_cost = est_tokens × (blended_rate / 1000)
132
+ ```
133
+
134
+ Round `est_tokens` to the nearest integer; format `est_cost` to 4 decimal places (e.g., `$0.0444`).
135
+
136
+ **Step 4 — Extract actual tokens per assessment agent.** For each completed assessment agent, read its `<usage>` block from its result:
137
+
138
+ - If the block is present: `actual_tokens = input_tokens + output_tokens`. Compute `actual_cost = actual_tokens × (blended_rate / 1000)` at 4 decimal places.
139
+ - If the block is missing or unparseable: set `actual_tokens = "N/A"` and `actual_cost = "N/A"`. Log a warning: `"[agent_name] produced no <usage> block; token data unavailable"`. Exclude this agent from phase subtotal and grand total calculations.
140
+
141
+ Status remains `"complete"` in both cases — a missing usage block does not indicate agent failure.
142
+
143
+ **Step 5 — Compute Phase 3 subtotals.** Sum `est_tokens` and `est_cost` across all assessment agents (all rows, regardless of N/A status). Sum `actual_tokens` and `actual_cost` only across rows where actual values are numeric (exclude N/A rows). Format subtotal costs to 2 decimal places.
144
+
145
+ **Step 6 — Write `{PREFIX}-Token-Estimate.md`.** Write the complete file at `docs/assessments/{PREFIX}/{PREFIX}-Token-Estimate.md` using the template below. Replace all `{PREFIX}`, `{N}`, and `{placeholder}` markers with computed values.
146
+
147
+ ```markdown
148
+ # Token Estimate — {PREFIX} — Assessment Pipeline
149
+
150
+ > Estimates computed before execution. Actuals accumulated as agents complete.
151
+ > Assessment agents (Phase 3) actuals: filled at end of phase.
152
+ > Intervention documentation (Phase 4) actuals: filled on completion.
153
+ > Orchestrator row added by assess-codebase skill at pipeline end.
154
+ > Pricing model: docs/pricing.md (80% input / 20% output split).
155
+
156
+ ## Estimation model
157
+
158
+ | Parameter | Value |
159
+ |-----------|-------|
160
+ | Avg chars per token | 4 |
161
+ | Haiku system prompt | ~2,000 tokens |
162
+ | Sonnet system prompt | ~3,000 tokens |
163
+ | Base overhead per call | ~5,000 tokens |
164
+ | Input/output split | 80% / 20% |
165
+
166
+ ## Assessment agents (Phase 3)
167
+
168
+ | Agent | Model | Est. tokens | Est. cost ($) | Actual tokens | Actual cost ($) | Status |
169
+ |-------|-------|-------------|---------------|---------------|-----------------|--------|
170
+ {one row per assessment agent that ran — computed values from Steps 3 and 4 — status = "complete"}
171
+
172
+ ## Intervention documentation (Phase 4)
173
+
174
+ > To be populated after Phase 4 completion.
175
+
176
+ | Agent | Model | Est. tokens | Est. cost ($) | Actual tokens | Actual cost ($) | Status |
177
+ |-------|-------|-------------|---------------|---------------|-----------------|--------|
178
+
179
+ ## Remediation
180
+
181
+ Remediation effort is tracked separately via the feature delivery pipeline (not part of this assessment).
182
+
183
+ ## Phase subtotals
184
+
185
+ | Phase | Est. tokens | Est. cost ($) | Actual tokens | Actual cost ($) |
186
+ |-------|-------------|---------------|---------------|-----------------|
187
+ | Assessment (Phase 3) | {est_tokens_total} | ${est_cost_total} | {actual_tokens_total} | ${actual_cost_total} |
188
+ | Intervention documentation (Phase 4) | — | — | — | — |
189
+
190
+ ## Grand total
191
+
192
+ > Partial — updated at pipeline end.
193
+
194
+ | Metric | Estimated | Actual |
195
+ |--------|-----------|--------|
196
+ | Total tokens | {est_tokens_total} | partial — updated at pipeline end |
197
+ | Total cost ($) | ${est_cost_total_2dp} | partial — updated at pipeline end |
198
+ ```
199
+
200
+ After writing, verify the file is readable: attempt to read it back. If it is unreadable, log an error: `"[timestamp] ERROR: Token Estimate file unreadable after write — {path}: {error}"`. If the write itself fails (e.g., permission denied, disk full), log the error: `"[timestamp] ERROR: Failed to write Token Estimate file — {path}: {error}"` and continue — the pipeline does not halt due to a Token Estimate file I/O failure.
201
+
202
+ **Step 7 — Log the file write.** Append to the process log:
203
+
204
+ ```
205
+ [timestamp] Phase 3 end: wrote Token Estimate file
206
+ Assessment agents: {N} rows ({completed} complete, {na_count} with N/A actuals)
207
+ Location: docs/assessments/{PREFIX}/{PREFIX}-Token-Estimate.md
208
+ ```
209
+
210
+ Where `{N}` is the total number of assessment agents that ran, `{completed}` is the count with numeric actual tokens, and `{na_count}` is the count with `N/A` actual tokens.
211
+
212
+ ### Write initial Effort Estimate file
213
+
214
+ **Step 1 — Read process log timestamps.** For each completed Phase 3 assessment agent, scan `{PREFIX}-process-log.txt` for lines matching:
215
+ - `Agent START: {agent_name}` → extract start timestamp
216
+ - `Agent DONE: {agent_name}` → extract end timestamp
217
+
218
+ If both timestamps are present, compute `actual_duration = end_timestamp − start_timestamp`, rounded to the nearest minute. Express as `"Xmin"` or `"Xh Ymin"` (e.g., `"12min"`, `"1h 5min"`). If either timestamp is missing, set `actual_duration = "N/A"` and log a warning: `"[agent_name] has no timestamp in the process log; actual duration unavailable"`.
219
+
220
+ For the batch wall-clock:
221
+ - `batch_start = minimum of all Phase 3 agent start timestamps`
222
+ - `batch_end = maximum of all Phase 3 agent end timestamps`
223
+ - `batch_actual = batch_end − batch_start`, rounded to nearest minute
224
+
225
+ **Step 2 — Compute estimates.** On the first run (no prior `{PREFIX}-Effort-Estimate.md` exists), set `est_duration = "N/A"` for all agent rows. On subsequent runs, extract the `actual_duration` from the previous run's rows for each agent and use as the new `est_duration`. If the prior file has `"N/A"` for an agent, continue using `"N/A"` as the estimate.
226
+
227
+ **Step 3 — Compute delta.** `delta = actual_duration − est_duration`. If either is `"N/A"`, set `delta = "N/A"`. For numeric values, express as `"-Xmin"` (faster), `"+Xmin"` (slower), or `"0min"` (on target).
228
+
229
+ **Step 4 — Write `{PREFIX}-Effort-Estimate.md`.** Write to `docs/assessments/{PREFIX}/{PREFIX}-Effort-Estimate.md`. Create the directory if it does not exist.
230
+
231
+ ```markdown
232
+ # Effort Estimate — {PREFIX} — Assessment Pipeline
233
+
234
+ > Wall-clock effort tracking for the assessment pipeline.
235
+ > Assessment agent durations: filled at end of Phase 3.
236
+ > Intervention documentation duration: filled at end of Phase 4.
237
+ > Remediation effort: estimated from Interventions Index at end of Phase 4 using fixed rates.
238
+ > Actual remediation effort tracked by feature delivery pipeline per intervention.
239
+ > Effort rates: CRITICAL=8h, HIGH=4h, MEDIUM=2h, LOW=1h (human hours, sequential).
240
+
241
+ ## Assessment phase
242
+
243
+ | Agent | Est. duration | Actual duration | Delta | Status |
244
+ |-------|--------------|-----------------|-------|--------|
245
+ | {agent_name} | {est_duration} | {actual_duration} | {delta} | {status} |
246
+ ... (one row per Phase 3 agent — intervention-documentation-standard appended at Phase 4 end)
247
+
248
+ ## Assessment phase subtotal
249
+
250
+ | Metric | Estimated | Actual | Delta |
251
+ |--------|-----------|--------|-------|
252
+ | Phase 3 assessment batch | {est_batch} | {actual_batch} | {delta_batch} |
253
+ | intervention-documentation-standard | {est} | {actual} | {delta} |
254
+ | Total | {total_est} | {total_actual} | {total_delta} |
255
+
256
+ > Note: Phase 3 "Actual" row uses batch wall-clock (max end − min start), not sum of individual durations.
257
+ > Individual agent durations are in the assessment phase table above.
258
+
259
+ ## Remediation effort estimate
260
+
261
+ > Pending intervention documentation completion.
262
+ ```
263
+
264
+ If a scope filter was applied, append to the header block: `> Note: Scope filter applied ({scope}). This estimate reflects only {scope} assessment areas.`
265
+
266
+ After writing, verify the file is readable. If unreadable, log an error: `"[timestamp] ERROR: Effort Estimate file unreadable after write — {path}: {error}"`. If the write fails, log: `"[timestamp] ERROR: Failed to write Effort Estimate file — {path}: {error}"` and continue — the pipeline does not halt due to a file I/O failure.
267
+
268
+ **Step 5 — Log the write.** Append to the process log:
269
+
270
+ ```
271
+ [timestamp] Phase 3 end: wrote Effort Estimate file
272
+ Assessment agents: {N} rows ({completed} with actuals, {na_count} with N/A)
273
+ Location: docs/assessments/{PREFIX}/{PREFIX}-Effort-Estimate.md
274
+ ```
275
+
108
276
  ---
109
277
 
110
278
  ## Phase 4 — Intervention Document Generation
@@ -113,90 +281,122 @@ Invoke `intervention-documentation-standard` with all produced assessment files
113
281
  - `{PREFIX}-INT-NNN-{slug}.md` per intervention
114
282
  - `{PREFIX}-Interventions-Index.md` (ordered by criticality and dependency)
115
283
 
116
- ---
284
+ ### Append intervention-documentation row to Token Estimate file
117
285
 
118
- ## Phase 5 — Remediation Gate
286
+ After `intervention-documentation-standard` completes, update the Token Estimate file as follows.
119
287
 
120
- Follow `docs/procedures/assessment-approval-gate.md`. If user selects "Assessment only", skip to Phase 8 (Summary).
288
+ **Step 1 — Extract actual tokens.** Read the `<usage>` block from the agent result: `actual_tokens = input_tokens + output_tokens`. Compute `actual_cost = actual_tokens × (blended_rate / 1000)` at 4 decimal places. If the block is missing or unparseable: set `actual_tokens = "N/A"` and `actual_cost = "N/A"`, and log a warning: `"intervention-documentation-standard produced no <usage> block; token data unavailable"`. Status remains `"complete"` regardless.
121
289
 
122
- ---
290
+ **Step 2 — Estimate tokens.** Apply the standard formula:
291
+
292
+ ```
293
+ est_tokens = base_overhead (5,000) + system_prompt_weight (3,000 for sonnet) + (intervention_doc_size_bytes / 4)
294
+ ```
123
295
 
124
- ## Phase 6 — Remediation Implementation
296
+ If the total size of produced intervention documents is unknown at estimation time, use `~15,500` tokens as a typical estimate. Compute `est_cost = est_tokens × (blended_rate / 1000)` at 4 decimal places. If pricing data is unavailable, set both cost fields to `N/A`.
297
+
298
+ **Step 3 — Replace the placeholder section.** Open `{PREFIX}-Token-Estimate.md`. Find the "Intervention documentation (Phase 4)" section (the `> To be populated after Phase 4 completion.` placeholder row). Replace the empty table body with the actual row:
299
+
300
+ | Agent | Model | Est. tokens | Est. cost ($) | Actual tokens | Actual cost ($) | Status |
301
+ |-------|-------|-------------|---------------|---------------|-----------------|--------|
302
+ | intervention-documentation-standard | sonnet | {est_tokens} | {est_cost} | {actual_tokens} | {actual_cost} | complete |
303
+
304
+ Remove the `> To be populated after Phase 4 completion.` note once the row is written.
305
+
306
+ **Step 4 — Log the completion.** Append to the process log:
307
+
308
+ ```
309
+ [timestamp] Phase 4 end: appended intervention-documentation-standard row to Token Estimate file. Actual tokens: {N} | Cost: ${X.XXXX}
310
+ ```
311
+
312
+ ### Update Effort Estimate file
313
+
314
+ **Step 1 — Read timestamps and compute duration.** Scan `{PREFIX}-process-log.txt` for `Agent START: intervention-documentation-standard` and `Agent DONE: intervention-documentation-standard`. Compute `actual_duration` (same formula as Phase 3 agents). If missing, set `actual_duration = "N/A"` and log a warning.
315
+
316
+ **Step 2 — Append intervention-documentation-standard row.** Open `{PREFIX}-Effort-Estimate.md` and append one row to the "Assessment phase" table:
125
317
 
126
- ### 6a. Create remediation branch
127
318
  ```
128
- git checkout -b feature/{PREFIX}-remediation
319
+ | intervention-documentation-standard | {est_duration} | {actual_duration} | {delta} | complete |
129
320
  ```
130
- Branch from `develop`.
131
321
 
132
- ### 6b. Build remediation execution plan
322
+ `est_duration` follows the same first-run / subsequent-run logic as Phase 3 agents.
323
+
324
+ **Step 3 — Update assessment phase subtotal.** Fill in the `intervention-documentation-standard` row in the subtotal table and compute the Total row values.
325
+
326
+ **Step 4 — Build remediation section.** Read `{PREFIX}-Interventions-Index.md` to extract intervention counts by severity. Apply fixed rates: CRITICAL × 8h, HIGH × 4h, MEDIUM × 2h, LOW × 1h. Sum for the human sequential total. If the Interventions Index is missing, write `"Interventions Index not found — remediation estimate unavailable"` as the remediation section body, log a warning, and continue. If all counts are 0, add: `> Note: No remediation required — zero findings identified.`
133
327
 
134
- Read `{PREFIX}-Interventions-Index.md` for approved interventions, dependency order, and parallelism opportunities.
328
+ **Step 5 — Replace placeholder.** Replace the `> Pending intervention documentation completion.` line in the "Remediation effort estimate" section with:
135
329
 
136
- ### 6c. Agent assignment
330
+ ```markdown
331
+ ## Remediation effort estimate
137
332
 
138
- Map each intervention to an agent via its `Suggested Agent Assignment` section. Prefer specialised agents (`security-hardening`, `god-class-decomposition`, etc.) over generic `developer-backend`. Fall back to `developer-backend` for uncovered backend/infra interventions.
333
+ > Derived from {PREFIX}-Interventions-Index.md. Rates: CRITICAL=8h, HIGH=4h, MEDIUM=2h, LOW=1h (human hours, sequential).
334
+ > Actual remediation effort tracked by feature delivery pipeline per intervention.
139
335
 
140
- ### 6d. Execution loop
336
+ | Severity | Count | Rate | Subtotal |
337
+ |----------|-------|------|---------|
338
+ | CRITICAL | {count} | 8h | {subtotal} |
339
+ | HIGH | {count} | 4h | {subtotal} |
340
+ | MEDIUM | {count} | 2h | {subtotal} |
341
+ | LOW | {count} | 1h | {subtotal} |
342
+ | **Total** | **{total_count}** | — | **{total_hours}h** |
141
343
 
142
- For each approved intervention (respecting dependency order):
143
- 1. Dispatch assigned agent with: intervention document path + codebase path
144
- 2. Wait for completion
145
- 3. Invoke `review-solution` on all changed files
146
- 4. **PASS** (no CRITICAL): commit — `git commit -m "fix({PREFIX}): {INT-NNN} — {title}"`
147
- 5. **FAIL** (CRITICAL): one rework cycle, re-review. After 2 failed cycles: escalate to user
148
- 6. **WARNING/INFO**: log to Issues Register (see `docs/procedures/issues-register.md`)
344
+ Human sequential total: {total_hours}h
345
+ ```
346
+
347
+ **Step 6 — Validate severity count.** Assert `sum(CRITICAL + HIGH + MEDIUM + LOW)` equals total intervention count in `{PREFIX}-Interventions-Index.md`. If they differ, log: `"Severity count mismatch between Effort Estimate and Interventions Index — check {PREFIX}-Interventions-Index.md"`. Use the Interventions Index counts as the authoritative source.
348
+
349
+ **Step 7 — Log the update.** Append to the process log:
350
+
351
+ ```
352
+ [timestamp] Phase 4 end: updated Effort Estimate with intervention-documentation-standard row
353
+ Actual duration: {X}min
354
+ [timestamp] Phase 4 end: populated remediation effort section
355
+ CRITICAL: {N}, HIGH: {N}, MEDIUM: {N}, LOW: {N}
356
+ Human sequential total: {X}h
357
+ Location: docs/assessments/{PREFIX}/{PREFIX}-Effort-Estimate.md
358
+ ```
149
359
 
150
360
  ---
151
361
 
152
- ## Phase 7 — Pull Request
153
-
154
- 1. Push: `git push -u origin feature/{PREFIX}-remediation`
155
- 2. Create PR targeting `develop`:
156
- ```
157
- gh pr create --title "fix({PREFIX}): codebase remediation" --body "$(cat <<'EOF'
158
- ## Assessment Summary
159
- - Assessment: {PREFIX} — {codebase description}
160
- - Total findings: N CRITICAL, N HIGH, N MEDIUM, N LOW
161
- - Source: docs/assessments/{PREFIX}/
162
-
163
- ## Remediation
164
- - Interventions approved: N / N proposed
165
- - Interventions implemented: N
166
- - Issues Register: {N} fixed, {N} deferred → see {PREFIX}-Issues.md
167
-
168
- ## Test plan
169
- - [ ] Build passes
170
- - [ ] Test suite passes
171
- - [ ] Manual verification of critical fixes
172
- EOF
173
- )"
174
- ```
362
+ ## Phase 5 — Findings Gate
363
+
364
+ Follow `docs/procedures/assessment-findings-gate.md`. The gate has two mandatory steps:
365
+
366
+ **Step 5a — Acknowledge:** present findings summary (including effort estimate) and wait for acknowledgement. Accept any non-empty text reply.
367
+
368
+ **Step 5b — Flag interventions:** prompt for INT-NNN identifiers to flag for feature delivery. Validate each against `{PREFIX}-Interventions-Index.md`; re-prompt if any are unknown. Accept "None" for zero flagged.
369
+
370
+ After both steps complete, write `{PREFIX}-Approvals.md` per the format in `docs/procedures/assessment-findings-gate.md`. Verify the file was written before proceeding to Phase 6.
371
+
372
+ No writes to the Effort Estimate file during Phase 5.
175
373
 
176
374
  ---
177
375
 
178
- ## Phase 8 — Summary
376
+ ## Phase 6 — Summary
377
+
378
+ Report:
179
379
 
180
380
  ```
181
- 📦 Assessment Manager — Run Summary
381
+ Assessment Manager — Run Summary
182
382
  ─────────────────────────────────────────────────────
183
383
  Target: {codebase path} | Prefix: {PREFIX}
184
384
  ─────────────────────────────────────────────────────
185
385
  Assessment:
186
- ✅ generic-software-assessment → {PREFIX}-Generic-Assessment.md
187
- ✅ layered-architecture-assessment → {PREFIX}-Layer-Assessment.md
188
- ✅ concurrency-safety-assessment → {PREFIX}-Concurrency-Assessment.md
386
+ ✅ {agent_1} → {PREFIX}-{output}.md
387
+ ✅ {agent_2} → {PREFIX}-{output}.md
388
+ ...
189
389
  ─────────────────────────────────────────────────────
190
390
  Findings: N CRITICAL | N HIGH | N MEDIUM | N LOW
191
- Interventions: N proposed | N approved
391
+ Interventions: N proposed | N flagged for feature delivery
192
392
  ─────────────────────────────────────────────────────
193
- Remediation:
194
- INT-001 ✅ PASS | INT-002 ✅ PASS | INT-003 ⏭ Deferred
195
- Issues: N fixed | N deferred → {PREFIX}-Issues.md
393
+ Approvals: {PREFIX}-Approvals.md
394
+ Effort Estimate: {PREFIX}-Effort-Estimate.md
395
+ Token Estimate: {PREFIX}-Token-Estimate.md
396
+ Process log: docs/assessments/{PREFIX}/{PREFIX}-process-log.txt
196
397
  ─────────────────────────────────────────────────────
197
- Pull Request: 🔗 {PR URL}
198
- Token usage: {N} tokens | Est. cost: $N.NN (see {PREFIX}-Token-Estimate.md)
199
- Process log: docs/assessments/{PREFIX}/{PREFIX}-process-log.txt
398
+ Flagged interventions can be actioned via /define-feature
399
+ referencing the INT-NNN document.
200
400
  ─────────────────────────────────────────────────────
201
401
  ```
202
402
 
@@ -204,15 +404,13 @@ Process log: docs/assessments/{PREFIX}/{PREFIX}-process-log.txt
204
404
 
205
405
  ## Guidelines
206
406
 
207
- - **Read all procedures at startup** — `process-log.md`, `issues-register.md`, `assessment-approval-gate.md`
208
- - **Track tokens for every agent call** — `<usage>` block → ledger → log
407
+ - **Read all procedures at startup** — `process-log.md`, `issues-register.md`, `assessment-findings-gate.md`, `token-estimation.md`
408
+ - **Token tracking — maintain an internal ledger keyed by agent name.** For each completed agent call: extract the `<usage>` block (input_tokens, output_tokens, model), compute `actual_tokens = input_tokens + output_tokens`, and record the result. If the block is absent or unparseable, record `actual_tokens = "N/A"` and log a warning — never halt because of a missing usage block. Estimate tokens for each agent BEFORE dispatch using `est_tokens = base_overhead (5,000) + system_prompt_weight (3,000 for sonnet, 2,000 for haiku) + (input_size_chars / 4)` and compute blended cost using the formula from `docs/pricing.md`. Exclude N/A rows from all cost totals and accuracy statistics; include them in the file as visible rows.
209
409
  - **Assessment agents are read-only** — never modify source files during assessment phases
210
- - **Never skip the Remediation Gate** — remediation cannot start without explicit approval
410
+ - **Never skip the Findings Gate** — it is mandatory regardless of finding count; `--force` re-presents it, never bypasses it
411
+ - **Assessment pipeline is read-only** — no branch creation, no code changes, no commits, no PRs
211
412
  - **Assessment agents run in parallel** — they have no mutual dependencies
212
- - **Prefer specialised remediation agents** over generic developer agents when available
213
413
  - **Plan before executing** — always show the plan first
214
414
  - **Fail fast on missing target** — if the target path does not exist, stop immediately
215
415
  - **Partial assessments are acceptable** — if one agent fails, continue with others
216
- - **Commit per intervention** — immediately after it passes review
217
- - **PR targets `develop`** — never push directly; do NOT auto-merge
218
416
  - **All output documents in English**
@@ -52,7 +52,7 @@ state = {
52
52
 
53
53
  ### 2a. Cross-feature analysis
54
54
 
55
- Compare `feature.md` against each `registry` entry for OVERLAP, DEPENDENCY, or CONFLICT (see `docs/procedures/feature-registry.md`). Show the cross-feature report to the user. If conflicts are found, use `AskUserQuestion` before continuing. If `registry` is empty, output "No prior features — no cross-reference needed."
55
+ Compare `feature.md` against each `registry` entry for OVERLAP, DEPENDENCY, or CONFLICT (see `docs/procedures/feature-registry.md`). Show the cross-feature report to the user. If conflicts are found, output a hard-stop message and wait for a text reply before continuing. Do NOT use `AskUserQuestion` — it cannot be answered through the agent relay chain. If `registry` is empty, output "No prior features — no cross-reference needed."
56
56
 
57
57
  ### 2b. Build execution plan
58
58
 
@@ -301,7 +301,7 @@ To add a new agent: create `.claude/agents/my-agent.md` with a clear `descriptio
301
301
  - **Plan before executing** — always show the plan first
302
302
  - **Never re-run fresh agents** unless `--force`
303
303
  - **Parallelize where possible** — independent agents run concurrently
304
- - **Gates are hard stops** — use `AskUserQuestion`, wait for the user, write `{PREFIX}-Approvals.md`, verify it on disk. Never auto-approve, never assume, never skip
304
+ - **Gates are hard stops** — output the hard-stop text and wait for a text reply from the user; write `{PREFIX}-Approvals.md`, verify it on disk. Do NOT use `AskUserQuestion` — it cannot be answered through the agent relay chain. Never auto-approve, never assume, never skip
305
305
  - **Pre-condition check at Phase 6** — read `{PREFIX}-Approvals.md` and verify both gates ✅ before touching the codebase. If missing, return to the gate
306
306
  - **Review at US level** — not per task
307
307
  - **Commit per US** — `feat({PREFIX}): implement US-XX — {title}`
@@ -1,10 +1,18 @@
1
1
  ---
2
- description: "Assess Codebase — starts the full codebase assessment pipeline (parallel assessment → findings consolidation → intervention documents → approval gate → remediation → review → PR). Usage: /assess-codebase [path] [--scope=architecture,security,quality,concurrency,devops] [--force]"
2
+ description: "Assess Codebase — starts the codebase assessment pipeline (parallel assessment → findings consolidation → intervention documents → findings gate). Usage: /assess-codebase [path] [--scope=architecture,security,quality,concurrency,devops] [--force]"
3
3
  argument-hint: "[path] [--scope=architecture,security,quality,concurrency,devops] [--force]"
4
- disable-model-invocation: true
5
4
  ---
6
5
 
7
- Spawn the `assessment-manager` agent with the arguments provided by the user:
6
+ # Assess Codebase
7
+
8
+ Orchestrates the full codebase assessment pipeline by spawning the `assessment-manager` agent
9
+ and, once it completes, recording the real token consumption in a dedicated file.
10
+
11
+ ---
12
+
13
+ ## Step 1 — Spawn the Assessment Manager
14
+
15
+ Spawn the `assessment-manager` agent with the exact arguments the user provided:
8
16
 
9
17
  ```
10
18
  subagent_type: assessment-manager
@@ -13,4 +21,121 @@ prompt: <path> [--scope=...] [--force]
13
21
 
14
22
  If no path is provided, use `.` (current working directory).
15
23
 
16
- Pass the exact arguments the user typed after `/assess-codebase` as the prompt to the agent. Do not add anything else.
24
+ Wait for the agent to complete. Capture its full result, including the `<usage>` block
25
+ that appears at the end of the tool result (format: `subagent_tokens: N`).
26
+
27
+ The assessment-manager handles the full pipeline internally — it presents the Phase 5 Findings Gate
28
+ to the user, records the acknowledgement and flagged interventions, and ends at the summary.
29
+ The skill waits for the agent to complete and proceeds to Step 2 regardless of outcome.
30
+
31
+ ---
32
+
33
+ ## Step 2 — Extract orchestrator token usage
34
+
35
+ From the `<usage>` block of the assessment-manager result, read:
36
+ - `subagent_tokens` — total tokens consumed by the assessment-manager agent itself (input + output)
37
+ - `duration_ms` — wall-clock duration of the assessment-manager run
38
+
39
+ If the `<usage>` block is missing: log warning
40
+ `"assessment-manager produced no <usage> block; orchestrator token data unavailable"`
41
+ and set `actual_tokens = "N/A"`.
42
+
43
+ ---
44
+
45
+ ## Step 3 — Complete Token Estimate file
46
+
47
+ The assessment-manager will have written `{PREFIX}-Token-Estimate.md` at
48
+ `docs/assessments/{PREFIX}/{PREFIX}-Token-Estimate.md` with assessment and
49
+ intervention-documentation rows. Read that file, then **append** the following in order:
50
+
51
+ ### 3a — Orchestrator row
52
+
53
+ The orchestrator row is NOT added to any of the three phase-level tables (Assessment agents, Intervention documentation, Remediation agents). It appears only in the Actuals vs Estimate section (Step 3b below).
54
+
55
+ For use in Step 3b, compute the orchestrator row values as follows:
56
+
57
+ - Estimated tokens: 80,000 (baseline from estimation model)
58
+ - Estimated cost: $0.4320 — computed as `80,000 × $0.005400 / 1,000` using the sonnet blended
59
+ rate from `docs/pricing.md`. If `docs/pricing.md` is missing, use `"N/A"` for est_cost.
60
+ - Actual tokens: value from `subagent_tokens` in the `<usage>` block; `"N/A"` if block was missing
61
+ - Actual cost: `(actual_tokens / 1,000) × blended_rate_per_1k`, formatted to 4 decimal places; `"N/A"` if actual_tokens is `"N/A"` or pricing data is unavailable
62
+
63
+ ### 3b — Actuals vs Estimate section
64
+
65
+ Append a horizontal rule (`---`) followed by an Actuals vs Estimate section. Include one
66
+ row for every agent that ran: all assessment agents, intervention-documentation-standard,
67
+ and the assessment-manager orchestrator.
68
+
69
+ ```markdown
70
+ ## Actuals vs Estimate
71
+
72
+ | Agent | Task/Scope | Model | Est. tokens | Actual tokens | Delta | Est. cost ($) | Actual cost ($) | Duration |
73
+ |-------|------------|-------|-------------|---------------|-------|---------------|-----------------|----------|
74
+ | {agent} | {scope} | {model} | {est} | {actual} | {±delta} | {est_cost} | {actual_cost} | {Xmin Ys} |
75
+ ```
76
+
77
+ Column rules:
78
+ - **Delta**: `actual_tokens − est_tokens`; show as `+N` or `−N`. Show `"N/A"` if either
79
+ value is missing.
80
+ - **Duration**: convert `duration_ms` to `Xmin Ys` where available; use `"—"` where not.
81
+ - Rows with `"N/A"` actual tokens are still shown in the table but excluded from any
82
+ aggregate calculations.
83
+
84
+ ### 3c — Estimation accuracy by agent type (conditional)
85
+
86
+ If 2 or more distinct model tiers appear in the rows that have non-`"N/A"` actual tokens
87
+ (e.g., haiku AND sonnet), append:
88
+
89
+ ```markdown
90
+ ## Estimation accuracy by agent type
91
+
92
+ | Model | Count | Avg est. tokens | Avg actual tokens | Avg delta | Trend |
93
+ |-------|-------|-----------------|-------------------|-----------|-------|
94
+ | {model} | {N} | {avg_est} | {avg_actual} | {avg_delta} | {trend} |
95
+ ```
96
+
97
+ Column rules:
98
+ - One row per model tier (e.g., `haiku`, `sonnet`)
99
+ - Exclude rows with `"N/A"` actual tokens from all averages
100
+ - **Trend**: `"over-target"` if avg delta > +20% of avg est; `"under-target"` if avg delta
101
+ < −20% of avg est; otherwise `"on-target"`
102
+
103
+ ### 3d — Grand Total section
104
+
105
+ Update the Grand Total section already written in the file: replace the
106
+ `"partial — updated at pipeline end"` marker with `"Final"` and fill in the computed values:
107
+
108
+ ```markdown
109
+ ## Grand Total (Final)
110
+
111
+ | Metric | Estimated | Actual | Delta | Delta % |
112
+ |--------|-----------|--------|-------|---------|
113
+ | Total tokens (all agents) | {sum_est} | {sum_actual} | ±{delta} | {delta_pct} |
114
+ | Total cost ($) | ${sum_est_cost} | ${sum_actual_cost} | ±${delta_cost} | {delta_cost_pct} |
115
+ | Total wall-clock | — | {Xmin Ys} | — | — |
116
+ ```
117
+
118
+ Column rules:
119
+ - **Delta %**: `(delta / estimated) × 100` at 1 decimal place; show `"N/A"` if estimated is 0
120
+
121
+ Summation rules:
122
+ - Sum all agents' tokens and costs; exclude `"N/A"` rows from sums
123
+ - Cost values: 2 decimal places for subtotals and totals; 4 decimal places for per-row costs
124
+ - The "Remediation" section in the Token Estimate file contains a static note (not a placeholder); leave it as-is
125
+ - Wall-clock: convert `duration_ms` from the assessment-manager result to minutes/seconds
126
+
127
+ ---
128
+
129
+ ## Step 4 — Report to user
130
+
131
+ After writing to the Token Estimate file, report:
132
+
133
+ ```
134
+ Assessment pipeline complete.
135
+ Token estimate + actuals → {PREFIX}-Token-Estimate.md
136
+ Approvals → docs/assessments/{PREFIX}/{PREFIX}-Approvals.md
137
+ Process log → docs/assessments/{PREFIX}/{PREFIX}-process-log.txt
138
+ ```
139
+
140
+ If the Token Estimate file does not exist (assessment-manager failed before writing it):
141
+ note this in the report but do not halt — the skill has completed its work.
package/CHANGELOG.md CHANGED
@@ -1,2 +1,4 @@
1
+ ## 0.1.6 (2026-07-16)
2
+ ## 0.1.5 (2026-07-14)
1
3
  ## 0.1.4 (2026-07-14)
2
4
  ## 0.1.3 (2026-07-14)
package/CLAUDE.md CHANGED
@@ -114,7 +114,7 @@ Generic procedures live in `docs/procedures/` in this toolkit. Projects can over
114
114
 
115
115
  | Agent | subagent_type | Purpose |
116
116
  |-------|---------------|---------|
117
- | Assessment Manager | `assessment-manager` | Orchestrates the full assessment + remediation pipeline |
117
+ | Assessment Manager | `assessment-manager` | Orchestrates the assessment pipeline: parallel assessment → intervention docs → approval gate → effort/token estimates → summary |
118
118
  | Generic Software Assessment | `generic-software-assessment` | Broad quality analysis across architecture, security, testability, observability, DevOps |
119
119
  | Intervention Documentation Standard | `intervention-documentation-standard` | Generates structured, self-contained intervention documents from assessment findings |
120
120
  | God Class Decomposition | `god-class-decomposition` | Safe incremental decomposition of oversized classes and methods |
@@ -33,7 +33,7 @@ npx @dtlabs/ai-toolkit --local . --force
33
33
 
34
34
  ### Global installation
35
35
 
36
- Merges `.claude/agents`, `.claude/skills`, `docs/`, and `CLAUDE.md` into `~/.claude/`:
36
+ Merges `.claude/agents`, `.claude/skills`, `.claude/commands`, `docs/`, and `CLAUDE.md` into `~/.claude/`:
37
37
 
38
38
  ```bash
39
39
  npm install -g @dtlabs/ai-toolkit
@@ -68,13 +68,3 @@ ai-toolkit --global --force
68
68
  ```bash
69
69
  npm view @dtlabs/ai-toolkit version
70
70
  ```
71
-
72
- ---
73
-
74
- ## Publishing a new version
75
-
76
- ```bash
77
- npm login # authenticate with your npmjs.com account
78
- npm version patch # bump version (patch / minor / major)
79
- npm publish --access public # --access public is required for scoped packages
80
- ```
@@ -16,15 +16,19 @@ Show the user a summary of the generated documents:
16
16
 
17
17
  ### Step 2 — Request explicit sign-off
18
18
 
19
- Use `AskUserQuestion` with this exact phrasing:
19
+ Output the following hard-stop message and **wait for a text reply from the user**. Do NOT use `AskUserQuestion` — it cannot be answered through the agent relay chain.
20
20
 
21
- > **"Please review the documents above and choose one of the options below.
22
- > The pipeline CANNOT continue until you make a selection."**
21
+ ```
22
+ ⛔ GATE 1 — DOCS APPROVAL — HARD STOP
23
+
24
+ Please review the documents above and reply with one of:
23
25
 
24
- Options:
25
- - **"Approve — proceed to Work Breakdown"**
26
- - **"Request changes"** — describe what to change; the pipeline re-runs affected agents, re-validates, and returns here (max 2 revision cycles)
27
- - **"Approve with notes"** — accepted with comments recorded in the approval file
26
+ "Approve — proceed to Work Breakdown"
27
+ "Request changes: <describe what to change>"
28
+ "Approve with notes: <your comments>"
29
+
30
+ The pipeline CANNOT continue until you reply directly.
31
+ ```
28
32
 
29
33
  ### Step 3 — Write approval record BEFORE continuing
30
34
 
@@ -81,15 +85,19 @@ Show key metrics:
81
85
 
82
86
  ### Step 3 — Request explicit sign-off
83
87
 
84
- Use `AskUserQuestion`:
88
+ Output the following hard-stop message and **wait for a text reply from the user**. Do NOT use `AskUserQuestion` — it cannot be answered through the agent relay chain.
85
89
 
86
- > **"Please review the Work Breakdown above and choose one of the options below.
87
- > Implementation CANNOT start until you approve."**
90
+ ```
91
+ ⛔ GATE 2 — WORK BREAKDOWN APPROVAL — HARD STOP
92
+
93
+ Please review the Work Breakdown above and reply with one of:
88
94
 
89
- Options:
90
- - **"Approve — start implementation"**
91
- - **"Request changes"** — re-run `generate-work-breakdown` with feedback (max 2 cycles)
92
- - **"Approve with notes"** — accepted with comments recorded
95
+ "Approve — start implementation"
96
+ "Request changes: <describe what to change>"
97
+ "Approve with notes: <your comments>"
98
+
99
+ Implementation CANNOT start until you reply directly.
100
+ ```
93
101
 
94
102
  ### Step 4 — Update approval record BEFORE starting implementation
95
103
 
@@ -0,0 +1,120 @@
1
+ # Procedure: Assessment Findings Gate
2
+
3
+ After intervention documents are generated, present findings to the user and obtain explicit acknowledgement before closing the pipeline. The pipeline is read-only — no remediation code is written.
4
+
5
+ ## Gate presentation
6
+
7
+ ```
8
+ 📊 Assessment Complete — {PREFIX}
9
+ ═══════════════════════════════════════
10
+ Assessment results:
11
+ Generic quality: [KPI summary scores]
12
+ Layer violations: N CRITICAL, N HIGH, N MEDIUM
13
+ Concurrency hazards: N CRITICAL, N HIGH
14
+ [other assessments]
15
+
16
+ Interventions proposed: N total
17
+ CRITICAL: N (immediate attention)
18
+ HIGH: N (next sprint)
19
+ MEDIUM: N (planned cycle)
20
+ LOW: N (backlog)
21
+
22
+ Interventions Index: docs/assessments/{PREFIX}/{PREFIX}-Interventions-Index.md
23
+
24
+ Remediation Effort Estimate (from {PREFIX}-Effort-Estimate.md)
25
+ ─────────────────────────────────────────────────────────────
26
+ CRITICAL: N × 8h = Xh
27
+ HIGH: N × 4h = Xh
28
+ MEDIUM: N × 2h = Xh
29
+ LOW: N × 1h = Xh
30
+ Total estimated (human sequential): Xh
31
+ ─────────────────────────────────────────────────────────────
32
+ ```
33
+
34
+ ## Step 5a — Mandatory acknowledgement
35
+
36
+ Output the following hard-stop message and **wait for a text reply from the user**. Do NOT use `AskUserQuestion` — it cannot be answered through the agent relay chain.
37
+
38
+ ```
39
+ ⛔ FINDINGS GATE — STEP 1 OF 2 — HARD STOP
40
+
41
+ Please review the Interventions Index above.
42
+
43
+ Reply with "Acknowledged" (or any text) to confirm you have reviewed the findings.
44
+
45
+ The pipeline CANNOT continue until you reply directly.
46
+ ```
47
+
48
+ Accept any non-empty text reply as acknowledgement. Record the timestamp.
49
+
50
+ ## Step 5b — Optional intervention flagging
51
+
52
+ Immediately after acknowledgement, output:
53
+
54
+ ```
55
+ FINDINGS GATE — STEP 2 OF 2
56
+
57
+ Which interventions (if any) do you want to flag for feature delivery?
58
+
59
+ Reply with one of:
60
+ "None" — no interventions flagged
61
+ "INT-001, INT-003, ..." — comma-separated list of INT-NNN identifiers
62
+
63
+ Flagged interventions will be recorded in {PREFIX}-Approvals.md.
64
+ Any intervention can be actioned later via /define-feature referencing its INT-NNN document.
65
+ ```
66
+
67
+ **Validation:** For each INT-NNN the user provides, verify it exists in `{PREFIX}-Interventions-Index.md`. If any identifier is unknown, report it and re-prompt Step 5b without proceeding:
68
+
69
+ ```
70
+ Unknown identifier(s): INT-099
71
+ Please re-enter — only identifiers from {PREFIX}-Interventions-Index.md are valid.
72
+ ```
73
+
74
+ Re-prompt until all listed identifiers are valid, or the user replies "None".
75
+
76
+ ## Approvals file format
77
+
78
+ Write `{PREFIX}-Approvals.md` in the assessment directory **only after both steps complete successfully**. Never write a partial file.
79
+
80
+ ```markdown
81
+ # Findings Acknowledgement — {PREFIX}
82
+
83
+ ## Assessment Reviewed
84
+
85
+ | Document | Status | Date | Notes |
86
+ |----------|--------|------|-------|
87
+ | {PREFIX}-Generic-Assessment.md | Acknowledged | {date} | — |
88
+ | {PREFIX}-Interventions-Index.md | Acknowledged | {date} | — |
89
+
90
+ ## Interventions Flagged for Feature Delivery
91
+
92
+ | Intervention | Flagged | Date | Notes |
93
+ |---|---|---|---|
94
+ | INT-001 — {slug} | Yes | {date} | — |
95
+ | INT-002 — {slug} | No | {date} | Not selected |
96
+ ```
97
+
98
+ Rules:
99
+ - Every intervention from `{PREFIX}-Interventions-Index.md` must appear — one row per intervention
100
+ - "Flagged: Yes" for each INT-NNN the user explicitly named in Step 5b
101
+ - "Flagged: No" for all others, with Notes = "Not selected"
102
+ - If zero interventions were flagged, add a note below the table: `> No interventions selected for feature delivery.`
103
+ - If zero interventions exist (empty index), write the table header only and add: `> No interventions proposed — zero findings identified.`
104
+
105
+ ## Process log entries
106
+
107
+ ```
108
+ [timestamp] Phase 5 gate: presented findings (N interventions)
109
+ [timestamp] Phase 5 gate: Step 5a — user acknowledged findings
110
+ [timestamp] Phase 5 gate: Step 5b — user flagged: INT-001, INT-003 (or "none")
111
+ [timestamp] Phase 5 gate: wrote {PREFIX}-Approvals.md
112
+ ```
113
+
114
+ ## Rules
115
+
116
+ - **Never skip this gate** — it is mandatory regardless of finding count
117
+ - **Never write a partial Approvals file** — both steps must complete before writing
118
+ - **`--force` does not bypass the gate** — it re-presents the gate and overwrites the existing Approvals file
119
+ - **Log every gate event** in the process log (gate opened, step 5a complete, step 5b result, file written)
120
+ - **No remediation** — this gate never triggers branch creation, agent dispatch, commits, or PRs
@@ -10,6 +10,7 @@ Write `{PREFIX}-Token-Estimate.md` in the feature directory after Work Breakdown
10
10
  | Agent system prompt weight — haiku | ~2,000 tokens |
11
11
  | Agent system prompt weight — sonnet | ~3,000 tokens |
12
12
  | Base overhead per agent call | ~5,000 tokens |
13
+ | Input/output split | 80% / 20% |
13
14
 
14
15
  ## Cost estimation
15
16
 
@@ -98,10 +99,10 @@ The 80/20 input/output split is defined in `docs/pricing.md` — update there if
98
99
 
99
100
  ## Estimation accuracy by agent type
100
101
 
101
- | Agent type | Invocations | Avg est. tokens | Avg actual tokens | Avg delta | Trend |
102
- |-----------|------------|----------------|------------------|-----------|-------|
103
- | haiku agents | N | N | N | ±N | over/under/on-target |
104
- | sonnet agents | N | N | N | ±N | over/under/on-target |
102
+ | Model | Count | Avg est. tokens | Avg actual tokens | Avg delta | Trend |
103
+ |-------|-------|----------------|------------------|-----------|-------|
104
+ | haiku | N | N | N | ±N | ↑/↓/→ |
105
+ | sonnet | N | N | N | ±N | ↑/↓/→ |
105
106
 
106
107
  ## Grand Total
107
108
 
@@ -117,7 +118,38 @@ The 80/20 input/output split is defined in `docs/pricing.md` — update there if
117
118
  - One row per invocation — rework invocations get a "(rework)" suffix
118
119
  - Delta (tokens): negative = fewer tokens than estimated, positive = more
119
120
  - Cost: computed with blended formula from `docs/pricing.md`; use 4 decimal places for per-agent rows, 2 decimal places for totals
120
- - Trend: over-target if avg delta > +20%, under-target if < -20%, otherwise on-target
121
+ - Trend: "↑" if avg delta > +5% of avg est, "↓" if avg delta < -5% of avg est, otherwise "→"
121
122
  - Duration: convert `duration_ms` to minutes/seconds
122
123
  - Model: read from agent frontmatter, default to `sonnet` if not set
123
124
  - Missing `<usage>` block → write `N/A`, exclude from accuracy stats and cost totals
125
+
126
+ ## Assessment pipeline usage
127
+
128
+ The `assessment-manager` agent reads this procedure at Phase 3 start before dispatching assessment agents, in the same way that `project-manager` reads it before dispatching implementation agents. If this file is missing, the assessment pipeline halts with an error — this is the only non-graceful failure mode in the pipeline.
129
+
130
+ ### Assessment agent input size estimates
131
+
132
+ All assessment agents and the `intervention-documentation-standard` agent run on **sonnet** — use the `$0.005400/1k tokens` blended rate from `docs/pricing.md`.
133
+
134
+ Apply the standard formula with the code input sizes below:
135
+
136
+ ```
137
+ est_tokens = base_overhead + system_prompt_weight + (input_size_chars / 4)
138
+ = 5,000 + 3,000 (sonnet) + (input_size_chars / 4)
139
+ ```
140
+
141
+ | Agent | Approx. input size | Input tokens (chars ÷ 4) | Est. total tokens |
142
+ |-------|--------------------|--------------------------|-------------------|
143
+ | generic-software-assessment | ~80,000 chars of code | ~20,000 | ~28,000 |
144
+ | layered-architecture-assessment | ~50,000 chars of code | ~12,500 | ~20,500 |
145
+ | concurrency-safety-assessment | ~40,000 chars of code | ~10,000 | ~18,000 |
146
+ | other assessment agents | ~30,000 chars of code | ~7,500 | ~15,500 |
147
+ | intervention-documentation-standard | ~30,000 chars of assessment findings | ~7,500 | ~15,500 |
148
+
149
+ ### Remediation agents
150
+
151
+ Remediation agents receive an intervention document (~20,000 chars, ~5,000 tokens for the doc portion) plus relevant code sections. Use the intervention doc portion as the base input estimate and add code section tokens using the standard formula. Typical total range: 13,000–25,000 tokens depending on intervention scope.
152
+
153
+ ### Orchestrator (assessment-manager)
154
+
155
+ Estimate as the sum of all child agent result sizes (as input) plus process log writes (as output), following the same orchestrator rule as `project-manager`. Use **80,000 tokens** as a conservative baseline for a full pipeline run (assessment + intervention documentation + remediation phases combined).
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@dtlabs/ai-toolkit",
3
- "version": "0.1.4",
3
+ "version": "0.1.6",
4
4
  "description": "AI Toolkit — reusable agents and procedures for developers to build AI-powered applications.",
5
5
  "files": [
6
6
  "docs/",
@@ -1,60 +0,0 @@
1
- # Procedure: Assessment Remediation Gate
2
-
3
- After intervention documents are generated, present findings to the user and obtain explicit approval before any remediation code is written.
4
-
5
- ## Gate presentation
6
-
7
- ```
8
- 📊 Assessment Complete — ASSESS-001
9
- ═══════════════════════════════════════
10
- Assessment results:
11
- Generic quality: [KPI summary scores]
12
- Layer violations: N CRITICAL, N HIGH, N MEDIUM
13
- Concurrency hazards: N CRITICAL, N HIGH
14
- [other assessments]
15
-
16
- Interventions proposed: N total
17
- CRITICAL: N (immediate attention)
18
- HIGH: N (next sprint)
19
- MEDIUM: N (planned cycle)
20
- LOW: N (backlog)
21
-
22
- Interventions Index: docs/assessments/ASSESS-001/ASSESS-001-Interventions-Index.md
23
- ```
24
-
25
- Use `AskUserQuestion`:
26
- - **"Approve all"** — all proposed interventions approved
27
- - **"Approve CRITICAL only"** — only CRITICAL severity approved
28
- - **"Select interventions"** — user specifies INT-NNN to approve (comma-separated)
29
- - **"Assessment only"** — stop here, no remediation
30
-
31
- If the user selects **"Assessment only"**, skip to the Summary phase.
32
-
33
- ## Approvals file format
34
-
35
- Write `{PREFIX}-Approvals.md` in the assessment directory:
36
-
37
- ```markdown
38
- # Approval Record — {PREFIX}
39
-
40
- ## Assessment Approvals
41
-
42
- | Document | Status | Date | Notes |
43
- |----------|--------|------|-------|
44
- | {PREFIX}-Generic-Assessment.md | ✅ Reviewed | {date} | — |
45
- | {PREFIX}-Interventions-Index.md | ✅ Reviewed | {date} | — |
46
-
47
- ## Remediation Scope
48
-
49
- | Intervention | Approved | Date | Notes |
50
- |---|---|---|---|
51
- | INT-001 — sql-injection-hardening | ✅ Approved | {date} | — |
52
- | INT-002 — god-class-decomposition | ✅ Approved | {date} | — |
53
- | INT-003 — di-refactoring | ❌ Deferred | {date} | Out of current sprint scope |
54
- ```
55
-
56
- ## Rules
57
-
58
- - **Never skip this gate** — remediation cannot start without explicit approval
59
- - **Log the approval outcome** in the process log
60
- - **Record every decision** including deferred interventions and the reason