@brunosps00/dev-workflow 2.2.0 → 2.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (154) hide show
  1. package/.claude-plugin/plugin.json +1 -1
  2. package/CHANGELOG.md +121 -0
  3. package/README.md +25 -15
  4. package/lib/constants.js +39 -36
  5. package/lib/doctor.js +1 -1
  6. package/lib/init.js +8 -0
  7. package/lib/instruction-health.js +38 -0
  8. package/lib/post-update.js +2 -1
  9. package/lib/validate.js +3 -1
  10. package/package.json +1 -1
  11. package/scaffold/agent-registry.json +17 -1
  12. package/scaffold/agents/core/dw-finding-refuter.md +55 -0
  13. package/scaffold/config/routing.json +61 -23
  14. package/scaffold/en/agent-instructions.md +32 -137
  15. package/scaffold/en/commands/dw-analyze-project.md +2 -0
  16. package/scaffold/en/commands/dw-autopilot.md +36 -241
  17. package/scaffold/en/commands/dw-brainstorm.md +1 -1
  18. package/scaffold/en/commands/dw-bugfix.md +10 -35
  19. package/scaffold/en/commands/dw-claude-run.md +47 -45
  20. package/scaffold/en/commands/dw-codex-run.md +48 -45
  21. package/scaffold/en/commands/dw-context-budget.md +6 -14
  22. package/scaffold/en/commands/dw-copilot-run.md +47 -41
  23. package/scaffold/en/commands/dw-find-skills.md +3 -1
  24. package/scaffold/en/commands/dw-generate-pr.md +2 -2
  25. package/scaffold/en/commands/dw-goal.md +25 -5
  26. package/scaffold/en/commands/dw-help.md +3 -3
  27. package/scaffold/en/commands/dw-install-aws-skills.md +2 -0
  28. package/scaffold/en/commands/dw-install-azure-skills.md +2 -0
  29. package/scaffold/en/commands/dw-plan.md +15 -8
  30. package/scaffold/en/commands/dw-qa.md +3 -0
  31. package/scaffold/en/commands/dw-resume.md +5 -11
  32. package/scaffold/en/commands/dw-review.md +154 -10
  33. package/scaffold/en/commands/dw-run.md +47 -177
  34. package/scaffold/en/commands/dw-secure-audit.md +4 -0
  35. package/scaffold/en/commands/dw-skill-health.md +39 -0
  36. package/scaffold/en/commands/dw-triage.md +8 -0
  37. package/scaffold/en/commands/dw-update.md +18 -4
  38. package/scaffold/en/references/automode.md +71 -0
  39. package/scaffold/en/references/command-routing.md +54 -0
  40. package/scaffold/en/references/execution-contract.md +55 -0
  41. package/scaffold/en/references/invariants.md +88 -0
  42. package/scaffold/en/references/untrusted-input.md +69 -0
  43. package/scaffold/en/templates/constitution-template.md +1 -0
  44. package/scaffold/en/templates/frontend-quality-template.md +43 -0
  45. package/scaffold/en/templates/task-template.md +20 -40
  46. package/scaffold/en/templates/tasks-template.md +9 -6
  47. package/scaffold/en/templates/techspec-template.md +4 -0
  48. package/scaffold/en/templates/triage-record-template.md +5 -0
  49. package/scaffold/pt-br/agent-instructions.md +32 -137
  50. package/scaffold/pt-br/commands/dw-analyze-project.md +3 -1
  51. package/scaffold/pt-br/commands/dw-autopilot.md +36 -241
  52. package/scaffold/pt-br/commands/dw-brainstorm.md +1 -1
  53. package/scaffold/pt-br/commands/dw-bugfix.md +10 -35
  54. package/scaffold/pt-br/commands/dw-claude-run.md +47 -46
  55. package/scaffold/pt-br/commands/dw-codex-run.md +48 -45
  56. package/scaffold/pt-br/commands/dw-context-budget.md +6 -14
  57. package/scaffold/pt-br/commands/dw-copilot-run.md +47 -43
  58. package/scaffold/pt-br/commands/dw-find-skills.md +3 -1
  59. package/scaffold/pt-br/commands/dw-generate-pr.md +2 -2
  60. package/scaffold/pt-br/commands/dw-goal.md +25 -5
  61. package/scaffold/pt-br/commands/dw-help.md +3 -3
  62. package/scaffold/pt-br/commands/dw-install-aws-skills.md +2 -0
  63. package/scaffold/pt-br/commands/dw-install-azure-skills.md +2 -0
  64. package/scaffold/pt-br/commands/dw-plan.md +15 -8
  65. package/scaffold/pt-br/commands/dw-qa.md +3 -0
  66. package/scaffold/pt-br/commands/dw-resume.md +7 -11
  67. package/scaffold/pt-br/commands/dw-review.md +154 -10
  68. package/scaffold/pt-br/commands/dw-run.md +47 -177
  69. package/scaffold/pt-br/commands/dw-secure-audit.md +5 -1
  70. package/scaffold/pt-br/commands/dw-skill-health.md +39 -0
  71. package/scaffold/pt-br/commands/dw-triage.md +8 -0
  72. package/scaffold/pt-br/commands/dw-update.md +18 -4
  73. package/scaffold/pt-br/references/automode.md +72 -0
  74. package/scaffold/pt-br/references/command-routing.md +54 -0
  75. package/scaffold/pt-br/references/execution-contract.md +55 -0
  76. package/scaffold/pt-br/references/invariants.md +88 -0
  77. package/scaffold/pt-br/references/untrusted-input.md +70 -0
  78. package/scaffold/pt-br/templates/constitution-template.md +1 -0
  79. package/scaffold/pt-br/templates/frontend-quality-template.md +43 -0
  80. package/scaffold/pt-br/templates/task-template.md +20 -40
  81. package/scaffold/pt-br/templates/tasks-template.md +9 -6
  82. package/scaffold/pt-br/templates/techspec-template.md +4 -0
  83. package/scaffold/pt-br/templates/triage-record-template.md +5 -0
  84. package/scaffold/scripts/hooks/git-guardrails.mjs +12 -0
  85. package/scaffold/scripts/hooks/session-cost.mjs +122 -7
  86. package/scaffold/scripts/lib/workflow-contract.mjs +101 -0
  87. package/scaffold/skill-registry.json +35 -23
  88. package/scaffold/skills/dw-chaos-engineering/SKILL.md +126 -0
  89. package/scaffold/skills/dw-chaos-engineering/references/vector-catalog.md +43 -0
  90. package/scaffold/skills/dw-cli-run/SKILL.md +39 -215
  91. package/scaffold/skills/dw-cli-run/references/dispatch-tuning.md +14 -72
  92. package/scaffold/skills/dw-codebase-intel/SKILL.md +5 -39
  93. package/scaffold/skills/dw-codebase-intel/agents/intel-updater.md +1 -1
  94. package/scaffold/skills/dw-codebase-intel/references/agents-detail.md +7 -0
  95. package/scaffold/skills/dw-codebase-intel/references/files-in-dw-intel-detail.md +16 -0
  96. package/scaffold/skills/dw-codebase-intel/references/how-dw-intel-reads-this-detail.md +12 -0
  97. package/scaffold/skills/dw-codebase-intel/references/how-to-compose-the-typical-flow-detail.md +11 -0
  98. package/scaffold/skills/dw-council/SKILL.md +2 -17
  99. package/scaffold/skills/dw-council/references/integration-with-other-dw-commands-detail.md +7 -0
  100. package/scaffold/skills/dw-council/references/phase-2-tensions-and-rebuttals-detail.md +15 -0
  101. package/scaffold/skills/dw-debug-protocol/references/six-step-triage.md +1 -0
  102. package/scaffold/skills/dw-domain-modeling/SKILL.md +1 -1
  103. package/scaffold/skills/dw-execute-phase/SKILL.md +12 -32
  104. package/scaffold/skills/dw-execute-phase/agents/executor.md +15 -252
  105. package/scaffold/skills/dw-execute-phase/agents/plan-checker.md +4 -207
  106. package/scaffold/skills/dw-execute-phase/references/atomic-commits.md +8 -18
  107. package/scaffold/skills/dw-execute-phase/references/de-sloppify-pass-optional-cleanup-task-detail.md +12 -0
  108. package/scaffold/skills/dw-execute-phase/references/deviation-rules-during-execution-detail.md +8 -0
  109. package/scaffold/skills/dw-execute-phase/references/how-the-two-agents-compose-detail.md +7 -0
  110. package/scaffold/skills/dw-execute-phase/references/plan-verification.md +9 -155
  111. package/scaffold/skills/dw-execute-phase/references/wave-coordination.md +6 -97
  112. package/scaffold/skills/dw-git-discipline/SKILL.md +2 -2
  113. package/scaffold/skills/dw-grilling/SKILL.md +1 -1
  114. package/scaffold/skills/dw-incident-response/SKILL.md +1 -64
  115. package/scaffold/skills/dw-incident-response/references/the-five-phases-detail.md +66 -0
  116. package/scaffold/skills/dw-llm-eval/SKILL.md +6 -53
  117. package/scaffold/skills/dw-llm-eval/references/agent-tool-use-evaluation-detail.md +23 -0
  118. package/scaffold/skills/dw-llm-eval/references/anti-patterns-will-block-in-dw-review-code-only-detail.md +9 -0
  119. package/scaffold/skills/dw-llm-eval/references/bounded-decisions.md +76 -0
  120. package/scaffold/skills/dw-llm-eval/references/consistency-run-to-run-stability-detail.md +10 -0
  121. package/scaffold/skills/dw-llm-eval/references/integration-with-dev-workflow-commands-detail.md +6 -0
  122. package/scaffold/skills/dw-llm-eval/references/judge-calibration.md +5 -0
  123. package/scaffold/skills/dw-llm-eval/references/the-oracle-ladder-detail.md +15 -0
  124. package/scaffold/skills/dw-memory/SKILL.md +4 -50
  125. package/scaffold/skills/dw-memory/references/context-budget.md +8 -56
  126. package/scaffold/skills/dw-memory/references/promotion-decision-test-detail.md +30 -0
  127. package/scaffold/skills/dw-minimalism/SKILL.md +4 -5
  128. package/scaffold/skills/dw-review-rigor/SKILL.md +31 -88
  129. package/scaffold/skills/dw-review-rigor/references/composition-audit.md +160 -0
  130. package/scaffold/skills/dw-review-rigor/references/refutation-pass.md +124 -0
  131. package/scaffold/skills/dw-review-rigor/references/the-five-rules-detail.md +56 -0
  132. package/scaffold/skills/dw-simplification/SKILL.md +2 -55
  133. package/scaffold/skills/dw-simplification/references/pattern-recognition-detail.md +11 -0
  134. package/scaffold/skills/dw-simplification/references/the-five-rules-detail.md +48 -0
  135. package/scaffold/skills/dw-source-grounding/SKILL.md +21 -56
  136. package/scaffold/skills/dw-source-grounding/references/the-protocol-detect-fetch-implement-cite-detail.md +58 -0
  137. package/scaffold/skills/dw-testing-discipline/SKILL.md +25 -177
  138. package/scaffold/skills/dw-testing-discipline/references/agent-guardrails.md +9 -167
  139. package/scaffold/skills/dw-testing-discipline/references/core-rules.md +3 -3
  140. package/scaffold/skills/dw-testing-discipline/references/mutation-testing.md +19 -0
  141. package/scaffold/skills/dw-testing-discipline/references/patterns.md +3 -12
  142. package/scaffold/skills/dw-ui-discipline/SKILL.md +2 -59
  143. package/scaffold/skills/dw-ui-discipline/references/frontend-engineering.md +46 -0
  144. package/scaffold/skills/dw-ui-discipline/references/the-four-grounding-questions-detail.md +61 -0
  145. package/scaffold/skills/dw-verify/SKILL.md +26 -148
  146. package/scaffold/skills/dw-verify/references/verification-discovery-and-attribution.md +2 -2
  147. package/scaffold/skills/humanizer/SKILL.md +2 -1
  148. package/scaffold/skills/security-review/SKILL.md +6 -184
  149. package/scaffold/skills/security-review/references/do-not-flag-detail.md +49 -0
  150. package/scaffold/skills/security-review/references/modern-threats.md +5 -0
  151. package/scaffold/skills/security-review/references/quick-patterns-reference-detail.md +56 -0
  152. package/scaffold/skills/security-review/references/review-process-detail.md +112 -0
  153. package/scaffold/skills/vercel-react-best-practices/SKILL.md +1 -90
  154. package/scaffold/skills/vercel-react-best-practices/references/quick-reference-detail.md +92 -0
@@ -26,19 +26,7 @@ If the feature is fully deterministic (no LLM in the loop), use `dw-testing-disc
26
26
 
27
27
  ## The oracle ladder
28
28
 
29
- Five rungs, climb from CHEAPEST/STRICTEST to MOST EXPENSIVE/SUBJECTIVE. Always start at the bottom; only climb when the lower rung can't cover the case.
30
-
31
- | Rung | What it checks | Cost | When to use |
32
- |------|----------------|------|-------------|
33
- | 1. **Exact match** | `output === expected` | ~free | Structured outputs (function calls, JSON with stable shape, classifications) |
34
- | 2. **Schema validation** | Output matches JSON schema / type contract | ~free | Output shape matters; specific values vary |
35
- | 3. **Outcome state** | Side effect produced the expected change (DB row, file written, tool called) | cheap | Agents, tool-use, RAG with concrete answers |
36
- | 4. **LLM-as-judge** | A different model grades the output against a rubric | medium ($$$) | Subjective quality (helpfulness, tone, faithfulness) where no rule can decide |
37
- | 5. **Human review** | Domain expert scores | expensive | Calibration of rung 4; high-stakes outputs; edge cases |
38
-
39
- **Rule:** never reach for rung 4 before checking if rungs 1-3 can cover the case. Every rung up costs an order of magnitude more (latency, money, calibration effort) — and adds entropy.
40
-
41
- See `references/oracle-ladder.md` for examples per rung and the climbing decision tree.
29
+ For the oracle ladder, read `references/the-oracle-ladder-detail.md`. Load only when this part of the task applies.
42
30
 
43
31
  ## LLM-as-judge discipline (when rung 4 is needed)
44
32
 
@@ -68,14 +56,7 @@ See `references/reference-dataset.md` for case-design principles, sampling from
68
56
 
69
57
  ## Consistency (run-to-run stability)
70
58
 
71
- The oracle ladder measures whether an output is *correct*; it does not measure whether the feature is *stable*. Because LLMs are non-deterministic, a case that passes once may fail the next run. Measure it:
72
-
73
- - Re-run each case **N times** (default 3) against the same code + model + prompt.
74
- - Record `pass_rate` (e.g. `2/3`) and `consistency` (fraction agreeing with the majority verdict) per case in `runs/<YYYY-MM-DD>.jsonl`.
75
- - A case with pass_rate strictly between `1/N` and `(N-1)/N` is **flaky** — a signal, not a pass. A 2/3 pass hides a one-in-three production failure; investigate before shipping.
76
- - Sample at the **temperature the feature uses in production** — consistency measured at temp 0 lies about a temp-0.7 feature.
77
-
78
- Deterministic rungs (1–3) are usually stable; consistency matters most for rung 4 (LLM-as-judge) and agent trajectories. Record the run count `N` in the run log so later comparisons are apples-to-apples.
59
+ For consistency (run-to-run stability), read `references/consistency-run-to-run-stability-detail.md`. Load only when this part of the task applies.
79
60
 
80
61
  ## RAG evaluation
81
62
 
@@ -91,27 +72,7 @@ Precision alone misses hallucination. Faithfulness alone misses retrieval failur
91
72
 
92
73
  ## Agent / tool-use evaluation
93
74
 
94
- Two questions distinguish good agent eval from bad:
95
-
96
- ### Question 1: outcome or trajectory?
97
-
98
- | Approach | What it checks | Failure mode |
99
- |----------|---------------|--------------|
100
- | **Outcome-only** | Did the agent achieve the goal? Was the final state correct? | Misses "ghost actions" — agent did the right thing for the wrong reasons |
101
- | **Trajectory** | Did the agent take the expected sequence of steps / tool calls? | Punishes legitimate creativity — agent solved it via a different valid path |
102
-
103
- **Recommendation:** outcome-only with side-effect assertion as default. Trajectory match for cases where the path matters (e.g., "must call `get-user` before `update-user`").
104
-
105
- ### Question 2: which trajectory match mode?
106
-
107
- When trajectory matching IS the right call, four modes are available:
108
-
109
- - **Strict** — same tool calls, same order, same arguments. Use when both sequence and parameters are part of the contract.
110
- - **Unordered** — same tool calls, any order. Use when concurrent calls are valid.
111
- - **Subset** — actual trajectory contains a subset of reference calls. Use to enforce "don't exceed expected tool use" (frugality / cost).
112
- - **Superset** — actual contains all reference calls plus possibly more. Use when specific tools are mandatory but extras are acceptable.
113
-
114
- See `references/agent-eval.md` for examples and the decision tree.
75
+ For agent / tool-use evaluation, read `references/agent-tool-use-evaluation-detail.md`. Load only when this part of the task applies.
115
76
 
116
77
  ## Required reading by context
117
78
 
@@ -119,26 +80,18 @@ See `references/agent-eval.md` for examples and the decision tree.
119
80
  |------------|------|
120
81
  | Designing an eval suite for an AI feature | `references/oracle-ladder.md` (climb the ladder) |
121
82
  | Using LLM-as-judge | `references/judge-calibration.md` (mandatory before relying on it) |
83
+ | Decision with a fixed answer set (classify, route, score a rubric) | `references/bounded-decisions.md` — score the candidates instead of generating, and why the probability is not a confidence |
122
84
  | Building / curating a reference dataset | `references/reference-dataset.md` |
123
85
  | RAG-specific feature | `references/rag-metrics.md` |
124
86
  | Agent / tool-use feature | `references/agent-eval.md` |
125
87
 
126
88
  ## Anti-patterns (will block in `/dw-review --code-only`)
127
89
 
128
- - **LLM-as-judge without calibration evidence.** PR adds LLM-as-judge but the calibration Spearman score is missing or < 0.80. REJECTED.
129
- - **Same-model judge.** Judge model is the same as the system under test. REJECTED unless explicitly documented (and even then, results are suspect).
130
- - **Single-rung eval.** Feature ships with only LLM-as-judge; no rung 1-3 grounding. REJECTED — the cheap rungs catch the loud failures.
131
- - **Synthetic-only dataset.** No traceable production-failure source for any case. REJECTED — confirm at least 20% of cases come from real user inputs.
132
- - **"Looks good to me" QA.** No reference dataset, no metric, no rubric — just sampling output and calling it good. REJECTED.
133
- - **Coverage as metric.** Quoting "we tested 50 prompts" without saying what was measured. The number is meaningless without the metric.
134
- - **Single-run eval on a non-deterministic feature.** Each case run once at production temperature; a lucky pass reported as a pass. REJECTED — report `pass_rate` over N runs (see Consistency).
90
+ For anti-patterns (will block in `/dw-review --code-only`), read `references/anti-patterns-will-block-in-dw-review-code-only-detail.md`. Load only when this part of the task applies.
135
91
 
136
92
  ## Integration with dev-workflow commands
137
93
 
138
- - `/dw-plan tasks`: when the PRD has an AI feature requirement, an eval-plan subtask is mandatory. The task references this skill's oracle ladder.
139
- - `/dw-review --code-only`: AI feature PRs require a reference dataset + ≥2 oracle rungs (lower rungs FIRST). The constitution gate also applies — if the project has principles about AI feature reliability, they're enforced here.
140
- - `/dw-qa --ai`: new mode (when this skill is bundled) — runs the reference dataset against the current implementation, logs to `QA/logs/ai/<feature>-<date>.jsonl`, computes precision@k / faithfulness / outcome accuracy per the feature type.
141
- - `/dw-bugfix` when the bug is an AI failure mode (hallucination, tool misuse, classification error): adds the failing case to the reference dataset BEFORE fixing — the case is now a regression test forever.
94
+ For integration with dev-workflow commands, read `references/integration-with-dev-workflow-commands-detail.md`. Load only when this part of the task applies.
142
95
 
143
96
  ## When the discipline bends
144
97
 
@@ -0,0 +1,23 @@
1
+ ## Agent / tool-use evaluation
2
+
3
+ Two questions distinguish good agent eval from bad:
4
+
5
+ ### Question 1: outcome or trajectory?
6
+
7
+ | Approach | What it checks | Failure mode |
8
+ |----------|---------------|--------------|
9
+ | **Outcome-only** | Did the agent achieve the goal? Was the final state correct? | Misses "ghost actions" — agent did the right thing for the wrong reasons |
10
+ | **Trajectory** | Did the agent take the expected sequence of steps / tool calls? | Punishes legitimate creativity — agent solved it via a different valid path |
11
+
12
+ **Recommendation:** outcome-only with side-effect assertion as default. Trajectory match for cases where the path matters (e.g., "must call `get-user` before `update-user`").
13
+
14
+ ### Question 2: which trajectory match mode?
15
+
16
+ When trajectory matching IS the right call, four modes are available:
17
+
18
+ - **Strict** — same tool calls, same order, same arguments. Use when both sequence and parameters are part of the contract.
19
+ - **Unordered** — same tool calls, any order. Use when concurrent calls are valid.
20
+ - **Subset** — actual trajectory contains a subset of reference calls. Use to enforce "don't exceed expected tool use" (frugality / cost).
21
+ - **Superset** — actual contains all reference calls plus possibly more. Use when specific tools are mandatory but extras are acceptable.
22
+
23
+ See `agent-eval.md` for examples and the decision tree.
@@ -0,0 +1,9 @@
1
+ ## Anti-patterns (will block in `/dw-review --code-only`)
2
+
3
+ - **LLM-as-judge without calibration evidence.** PR adds LLM-as-judge but the calibration Spearman score is missing or < 0.80. REJECTED.
4
+ - **Same-model judge.** Judge model is the same as the system under test. REJECTED unless explicitly documented (and even then, results are suspect).
5
+ - **Single-rung eval.** Feature ships with only LLM-as-judge; no rung 1-3 grounding. REJECTED — the cheap rungs catch the loud failures.
6
+ - **Synthetic-only dataset.** No traceable production-failure source for any case. REJECTED — confirm at least 20% of cases come from real user inputs.
7
+ - **"Looks good to me" QA.** No reference dataset, no metric, no rubric — just sampling output and calling it good. REJECTED.
8
+ - **Coverage as metric.** Quoting "we tested 50 prompts" without saying what was measured. The number is meaningless without the metric.
9
+ - **Single-run eval on a non-deterministic feature.** Each case run once at production temperature; a lucky pass reported as a pass. REJECTED — report `pass_rate` over N runs (see Consistency).
@@ -0,0 +1,76 @@
1
+ # Bounded decisions — when the answer space is already known
2
+
3
+ A large share of model calls in a product do not need written text. The application already knows the
4
+ possible answers and only needs one of them chosen: route this ticket to billing, technical or account ·
5
+ is this message urgent · how severe is this incident on a 1-4 scale.
6
+
7
+ Asking a generative model to *write* that answer means generating tokens one at a time, waiting for all of
8
+ them, parsing the result, and handling the case where it came back malformed. None of that work is about
9
+ the decision.
10
+
11
+ When the answer space is fixed in advance, the model can **score the candidates instead of writing one**.
12
+ The decision comes out of a single forward pass, as a distribution over the options you defined, with no
13
+ tokens to parse and no malformed output to handle.
14
+
15
+ This is not a new technique — multiple-choice benchmarks have been scored this way for years. It is worth
16
+ writing down because product code rarely uses it, and because it comes with one trap that costs more than
17
+ the speed it saves.
18
+
19
+ ## The trap: a probability is not a confidence
20
+
21
+ Scoring returns numbers that look like probabilities. Treat them as calibrated confidence and you will make
22
+ bad decisions with the appearance of rigour.
23
+
24
+ Raw model probabilities are usually **badly calibrated**: a candidate holding 80% of the mass is not right
25
+ 80% of the time. Preference tuning (RLHF and its relatives) tends to make this worse, because it sharpens
26
+ mass onto preferred answers beyond what their reliability warrants.
27
+
28
+ **This is a different property from what `references/judge-calibration.md` requires.** That file asks for
29
+ Spearman ≥0.80 between judge scores and human scores — rank agreement, which answers "does the judge order
30
+ cases the way a human would". Calibration answers a separate question: "when it says 0.8, is it right 80%
31
+ of the time". A judge can pass one and fail the other. Both matter, and they are not substitutes.
32
+
33
+ Two honest ways to handle it:
34
+
35
+ 1. **Do not interpret the number as a frequency.** If all you need is a threshold to route or abstain, pick
36
+ the threshold empirically against your reference dataset and never publish the raw value as a
37
+ confidence. This is usually enough, and it is the cheapest correct answer.
38
+ 2. **Calibrate it post-hoc.** Fit a mapping from raw score to observed frequency on held-out cases
39
+ (Platt scaling, isotonic regression). Now the number means something, and you can report it.
40
+
41
+ Either way, the reference dataset is what makes the number trustworthy — not the model that produced it.
42
+
43
+ ## What this gives your eval
44
+
45
+ A generative judge returns a verdict. A scoring judge returns a verdict **and a distribution**, which lets
46
+ the eval measure something accuracy alone cannot: whether the system knows when it is unsure.
47
+
48
+ - Plot a reliability curve — bucket predictions by confidence, compare each bucket's confidence against its
49
+ observed accuracy.
50
+ - Track expected calibration error across releases. A model swap can hold accuracy steady while wrecking
51
+ calibration, and accuracy-only evals will not show it.
52
+ - Route the low-confidence tail to a human or a more expensive rung instead of accepting it silently. That
53
+ is the same escalation shape `references/oracle-ladder.md` describes between rungs.
54
+
55
+ ## Two ways to implement it
56
+
57
+ **Score with the model you already run.** No new model, no new infrastructure: prefill the prompt and read
58
+ the scores of the candidate answers. Cheapest to adopt.
59
+
60
+ Two things to get right. First, if two options share their opening tokens, a naive first-token comparison
61
+ picks between prefixes rather than between answers — score whole candidates and normalise by length, which
62
+ is what serving frameworks that expose a choice primitive do by default. Second, the calibration problem
63
+ above is at its worst here, because the model was tuned to sound confident.
64
+
65
+ **Use a model built for typed decisions.** Purpose-built decision models take state plus a typed question
66
+ and return a distribution directly, trained for calibrated output rather than for human preference. Better
67
+ calibrated by construction, at the cost of another model to run, serve and keep current.
68
+
69
+ The first is right when the decision rides alongside work a general model is already doing. The second
70
+ earns its keep at high volume, where the decision *is* the workload.
71
+
72
+ ## When not to use any of this
73
+
74
+ If the right answer is a sentence — an explanation, a summary, a written recommendation — the answer space
75
+ is not bounded and none of this applies. Forcing it into a fixed set does not make it cheaper; it makes it
76
+ wrong. Rung 4 with a calibrated generative judge remains the tool for subjective output.
@@ -0,0 +1,10 @@
1
+ ## Consistency (run-to-run stability)
2
+
3
+ The oracle ladder measures whether an output is *correct*; it does not measure whether the feature is *stable*. Because LLMs are non-deterministic, a case that passes once may fail the next run. Measure it:
4
+
5
+ - Re-run each case **N times** (default 3) against the same code + model + prompt.
6
+ - Record `pass_rate` (e.g. `2/3`) and `consistency` (fraction agreeing with the majority verdict) per case in `runs/<YYYY-MM-DD>.jsonl`.
7
+ - A case with pass_rate strictly between `1/N` and `(N-1)/N` is **flaky** — a signal, not a pass. A 2/3 pass hides a one-in-three production failure; investigate before shipping.
8
+ - Sample at the **temperature the feature uses in production** — consistency measured at temp 0 lies about a temp-0.7 feature.
9
+
10
+ Deterministic rungs (1–3) are usually stable; consistency matters most for rung 4 (LLM-as-judge) and agent trajectories. Record the run count `N` in the run log so later comparisons are apples-to-apples.
@@ -0,0 +1,6 @@
1
+ ## Integration with dev-workflow commands
2
+
3
+ - `/dw-plan tasks`: when the PRD has an AI feature requirement, an eval-plan subtask is mandatory. The task references this skill's oracle ladder.
4
+ - `/dw-review --code-only`: AI feature PRs require a reference dataset + ≥2 oracle rungs (lower rungs FIRST). The constitution gate also applies — if the project has principles about AI feature reliability, they're enforced here.
5
+ - `/dw-qa --ai`: new mode (when this skill is bundled) — runs the reference dataset against the current implementation, logs to `QA/logs/ai/<feature>-<date>.jsonl`, computes precision@k / faithfulness / outcome accuracy per the feature type.
6
+ - `/dw-bugfix` when the bug is an AI failure mode (hallucination, tool misuse, classification error): adds the failing case to the reference dataset BEFORE fixing — the case is now a regression test forever.
@@ -16,6 +16,11 @@ Process:
16
16
  4. Compute Spearman rank correlation between human scores and judge scores.
17
17
 
18
18
  **Target:** Spearman ≥0.80.
19
+
20
+ > **This is rank agreement, not probability calibration.** Spearman answers "does the judge order cases the
21
+ > way a human would". It says nothing about whether a judge reporting 0.8 is right 80% of the time — a
22
+ > judge can pass this bar and still be badly calibrated. If your judge emits a confidence or a probability
23
+ > and anything downstream reads it as one, see `references/bounded-decisions.md`.
19
24
  **Acceptable:** 0.70-0.80 with documented rationale (e.g., "subjective tone judgments inherently noisy").
20
25
  **Reject:** <0.70. The judge is not measuring what you think it's measuring.
21
26
 
@@ -0,0 +1,15 @@
1
+ ## The oracle ladder
2
+
3
+ Five rungs, climb from CHEAPEST/STRICTEST to MOST EXPENSIVE/SUBJECTIVE. Always start at the bottom; only climb when the lower rung can't cover the case.
4
+
5
+ | Rung | What it checks | Cost | When to use |
6
+ |------|----------------|------|-------------|
7
+ | 1. **Exact match** | `output === expected` | ~free | Structured outputs (function calls, JSON with stable shape, classifications) |
8
+ | 2. **Schema validation** | Output matches JSON schema / type contract | ~free | Output shape matters; specific values vary |
9
+ | 3. **Outcome state** | Side effect produced the expected change (DB row, file written, tool called) | cheap | Agents, tool-use, RAG with concrete answers |
10
+ | 4. **LLM-as-judge** | A different model grades the output against a rubric | medium ($$$) | Subjective quality (helpfulness, tone, faithfulness) where no rule can decide |
11
+ | 5. **Human review** | Domain expert scores | expensive | Calibration of rung 4; high-stakes outputs; edge cases |
12
+
13
+ **Rule:** never reach for rung 4 before checking if rungs 1-3 can cover the case. Every rung up costs an order of magnitude more (latency, money, calibration effort) — and adds entropy.
14
+
15
+ See `oracle-ladder.md` for examples per rung and the climbing decision tree.
@@ -87,7 +87,7 @@ Create `MEMORY.md` and `<N>_memory.md` on first use with the template below. Nev
87
87
 
88
88
  ## Workflow
89
89
 
90
- ### 1. Load before editing code
90
+ ### 1. Load relevant task memory on start or resume
91
91
  - Read `MEMORY.md` and the current task's `<N>_memory.md` **before** any code change.
92
92
  - Treat these as mandatory context for the run, not optional notes.
93
93
  - If the caller marks either file for compaction, apply the Compaction Rules (below) before continuing.
@@ -116,34 +116,7 @@ Create `MEMORY.md` and `<N>_memory.md` on first use with the template below. Nev
116
116
 
117
117
  ## Promotion Decision Test
118
118
 
119
- Before promoting an item from `<N>_memory.md` to `MEMORY.md`, ask:
120
-
121
- 1. Will another task need this to avoid a mistake or rediscovery?
122
- 2. Is this fact durable across multiple runs, not just the current execution?
123
- 3. Is this information NOT already obvious from the PRD, TechSpec, task files, or the repository itself?
124
-
125
- All three must be "yes" to promote. If any is "no", the item stays in task memory.
126
-
127
- ### Confidence signal
128
-
129
- Tag each durable decision with a confidence in `[0.3–0.9]` plus the tasks that confirmed it — `… — [confidence: 0.7; seen in tasks 1,3,5]`:
130
-
131
- - Confirmed in **≥2 of the last 3 tasks** with no contradiction → **≥0.7** (trust it; safe to act on without re-deriving).
132
- - Confirmed once, or inferred but not yet reused → **0.3–0.5** (tentative; gather more signal before relying).
133
- - Contradicted by a later task or the repo → lower it or drop the decision (see Error Handling).
134
-
135
- Confidence makes cross-task learning explicit and is the signal `/dw-learn` reads to promote high-confidence decisions into durable instincts, constitution principles, or rules.
136
-
137
- ### Belongs in shared memory
138
- - A discovered constraint affecting multiple tasks ("the Stripe API rate-limits to 100 req/s — batch operations must respect this")
139
- - A cross-cutting architectural decision made during implementation ("chose React Server Components for data fetching across the whole feature")
140
- - An open risk future tasks must account for ("migration depends on schema v3 which is not yet deployed to staging")
141
-
142
- ### Stays in task memory
143
- - Files touched during this task's implementation
144
- - Debugging steps taken to resolve a task-specific error
145
- - The current task's objective and acceptance criteria snapshot
146
- - A workaround applied only to the current task's scope
119
+ For promotion decision test, read `references/promotion-decision-test-detail.md`. Load only when this part of the task applies.
147
120
 
148
121
  ## Compaction Rules
149
122
 
@@ -157,26 +130,7 @@ When flagged for compaction, apply inline:
157
130
 
158
131
  ## Context Budget
159
132
 
160
- Memory is part of a broader **context budget** the agent must respect during execution. A blown budget degrades reasoning before any output is produced — the model starts missing requirements, drops constraints, and reverts to averaged-over-training answers. Full guidance: [`references/context-budget.md`](references/context-budget.md).
161
-
162
- **Targets:**
163
-
164
- - **Total active context:** under **40k tokens** (rough working budget across PRD + TechSpec + tasks + MEMORY.md + per-task memory + open files).
165
- - **Reserve:** at least **120k tokens** of headroom for actual reasoning, tool output, and the model's response stream.
166
- - **Hard ceiling per memory file:** `MEMORY.md` ≤ 6KB; `<N>_memory.md` ≤ 3KB. Past those, compact instead of growing.
167
-
168
- **Anti-co-load rules** (apply on every load):
169
-
170
- 1. Never load two PRD specs in the same context. If switching PRDs, drop the previous PRD's spec/techspec/tasks references from the active set.
171
- 2. Never load multiple archived `.dw/bugfixes/` SUMMARY.md files together — load only what's needed for the active fix or query.
172
- 3. Never load `.dw/intel/files.json` and `.dw/intel/deps.json` simultaneously when answering a single question — pick the primary per query shape (see `dw-codebase-intel/references/query-patterns.md`).
173
- 4. Never load a design proposal AND the prior design's full text — if comparing, summarize the prior into 5-10 lines.
174
-
175
- **Monitoring signal:**
176
-
177
- If the agent finds itself reading large files repeatedly or summarizing the same fact across multiple turns, that's a budget signal. Compact memory and explicitly drop unrelated loaded context before proceeding. Note this in `MEMORY.md` under Handoff Notes so the next task starts lean.
178
-
179
- This budget is doctrine, not a hard gate. No command currently rejects work for exceeding 40k. The discipline lives here because future sessions read this skill first.
133
+ For context budget, read `references/context-budget.md`. Load only when this part of the task applies.
180
134
 
181
135
  ## Error Handling
182
136
 
@@ -199,7 +153,7 @@ Ported from Compozy's `cy-workflow-memory` skill (`/tmp/compozy/.agents/skills/c
199
153
 
200
154
  - Paths are `.dw/spec/<prd-slug>/` instead of `.compozy/tasks/<name>/`.
201
155
  - Task-local file is `<N>_memory.md` next to `<N>_task.md` (mirrors the existing dev-workflow task layout).
202
- - Inline Compaction Rules (Compozy keeps them in `references/memory-guidelines.md`); the budget discipline was extracted to `references/context-budget.md` because it's about model behavior, not file management.
156
+ - Inline Compaction Rules (Compozy keeps them in its upstream memory-guidelines reference); the budget discipline was extracted to `references/context-budget.md` because it's about model behavior, not file management.
203
157
 
204
158
  Credit: Compozy project (https://github.com/compozy/compozy).
205
159
 
@@ -1,63 +1,15 @@
1
- # Context Budget — Discipline
1
+ # Context and memory budget
2
2
 
3
- A working context window is finite. Past about 40k tokens of active content, model output quality degrades — the model starts to miss requirements, lose constraints, blur details across sources, and revert to training-set averages. This file codifies the discipline that keeps active context lean across a long dev-workflow session.
3
+ Read when memory grows or repeated exploration obscures the active task. Context capacity and compaction behavior vary by host/model; do not assume a universal 40k active limit or 120k reserve. A model may expose telemetry; label estimates and unknown usage accurately.
4
4
 
5
- ## The numbers
5
+ Keep relevant requirements, constraints, decisions, pending work and evidence available. Read sections or query indexes when that answers the question. Comparing PRDs/designs or tracing source plus dependencies can require multiple sources; do not prohibit necessary co-loading. Avoid bulk-loading unrelated archived summaries or task memories.
6
6
 
7
- - **Target active load:** under **40k tokens** total — PRD, TechSpec, tasks, MEMORY.md, per-task memory, all open files together.
8
- - **Reasoning reserve:** **120k+ tokens** of headroom for tool output, model thinking, and response generation.
9
- - **Soft alert:** when active load crosses **30k tokens**, plan to compact before the next major task.
10
- - **Compaction threshold for `MEMORY.md`:** ≤ 6KB on disk.
11
- - **Compaction threshold for `<N>_memory.md`:** ≤ 3KB on disk.
7
+ Use concise durable memory: target MEMORY.md at most 6KB and per-task memory at most 3KB, moving detailed logs to linked artifacts. These are project maintenance targets, not model capacity claims or task-completion gates. Preserve unresolved decisions, acceptance criteria, session identity and evidence when compacting.
12
8
 
13
- These are doctrine, not enforced by any hard gate. The cost of breaking them is degraded output, not a command failure. Catch it yourself; the framework can't.
9
+ Repeated reads, growing summaries and confused task boundaries are reasons to inspect context use, not proof of exceeding a hidden token threshold. Update the checkpoint if context limits threaten continuity; otherwise continue authorized work. Do not promise to evict individual messages from a host's context.
14
10
 
15
- ## The anti-co-load rules
11
+ Use `../../dw-codebase-intel/references/query-patterns.md` for indexed queries when useful. Verification failures still require diagnosing actual code/environment; do not attribute them to context size without evidence.
16
12
 
17
- Each is a "do not load both simultaneously" pairing. Pick one; if you must consult the other, summarize it to 5–10 lines first and drop the full version.
13
+ ## Attribution
18
14
 
19
- 1. **Two PRDs at once.** Working on `prd-foo`, then user asks about `prd-bar`? Finish `prd-foo`'s active turn first, drop its spec/techspec/tasks from active context, then load `prd-bar`. Never carry both spec sets.
20
-
21
- 2. **Multiple archived bugfix SUMMARY.md files.** Bugfix history queries should go through `.dw/intel/bugfixes.json` (compact, indexed), not by reading 20 SUMMARY.md files. Read individual SUMMARY.md only when the user is acting on that specific fix.
22
-
23
- 3. **Files.json + deps.json simultaneously.** They overlap. Each query has a primary file per `dw-codebase-intel/references/query-patterns.md`. Pick the primary; consult the secondary only if the primary returns nothing.
24
-
25
- 4. **A design proposal AND its predecessor's full text.** When comparing two designs, summarize the older one to 5–10 lines (the deltas the new one addresses) and reference it that way. Don't carry both in full.
26
-
27
- 5. **Per-task memories from non-adjacent tasks.** Only the current task's `<N>_memory.md` and the immediately-previous one (for handoff) belong in active context. Older `<N>_memory.md` files are durable storage — read them when explicitly handing back to an old task, not as part of routine loading.
28
-
29
- ## The monitoring signal
30
-
31
- You will not see a token counter. You can't measure your own budget directly. But these are observable proxies that mean **you are over budget**:
32
-
33
- - You are reading the same file twice in the same session without it having changed.
34
- - You are summarizing the same fact across multiple turns to keep it in scope.
35
- - You are answering questions by re-deriving from the repo what a memory file would tell you in one line.
36
- - You are confusing details between two PRDs, tasks, or designs.
37
- - Your turn outputs are getting longer to compensate for context that is silently slipping out of the model's attention.
38
-
39
- When you notice any of these, stop:
40
-
41
- 1. Compact `MEMORY.md` and the active `<N>_memory.md` per the Compaction Rules in `SKILL.md`.
42
- 2. Explicitly state in chat what you are dropping from active context.
43
- 3. Note the budget event in `MEMORY.md` under Handoff Notes ("compacted at task N — see this section for what was dropped").
44
- 4. Resume the turn.
45
-
46
- ## What is NOT a budget problem
47
-
48
- These look like budget problems but aren't — don't compact for them:
49
-
50
- - **A genuinely complex task.** Some features legitimately need many files. Read what you need; the alternative (working from partial context) is worse.
51
- - **A long verification report.** Verify output can be long. Don't summarize it before logging it to disk; truncate the chat surface, not the artifact.
52
- - **A long user message.** The user wrote what they wrote. Don't compact to shave their words.
53
- - **The first PRD load of a new feature.** First-load is fine; second-load of the same PRD in the same turn is the signal.
54
-
55
- ## Integration with other skills
56
-
57
- - **`dw-memory` (this skill)**: compaction is the lever for memory files; this file describes when and why.
58
- - **`dw-codebase-intel`**: `query-patterns.md` defines the primary-vs-secondary rule that anti-co-load rule #3 references.
59
- - **`dw-verify`**: a budget-related degradation may show up as a verify regression. If verify keeps failing on facts the PRD already states, suspect budget before suspecting code.
60
-
61
- ## Source
62
-
63
- Adapted from [`tech-leads-club/agent-skills/tlc-spec-driven`](https://github.com/tech-leads-club/agent-skills/tree/main/packages/skills-catalog/skills/(development)/tlc-spec-driven) (CC-BY-4.0, Felipe Rodrigues). The 40k target, the anti-co-load principle, and the monitoring-signal framing come from there. The specific anti-co-load pairings, file-size ceilings, and the integration with `dw-memory`/`dw-codebase-intel` are dev-workflow-specific.
15
+ The earlier context-budget guidance adapted tech-leads-club's [tlc-spec-driven](https://github.com/tech-leads-club/agent-skills/tree/main/packages/skills-catalog/skills/(development)/tlc-spec-driven) (CC-BY-4.0, Felipe Rodrigues). This revision retains scoped retrieval and durable handoffs; rejects universal token thresholds and unconditional anti-co-loading rules. See docs/skills-ecosystem-comparison.md in the package source for the modernization decision.
@@ -0,0 +1,30 @@
1
+ ## Promotion Decision Test
2
+
3
+ Before promoting an item from `<N>_memory.md` to `MEMORY.md`, ask:
4
+
5
+ 1. Will another task need this to avoid a mistake or rediscovery?
6
+ 2. Is this fact durable across multiple runs, not just the current execution?
7
+ 3. Is this information NOT already obvious from the PRD, TechSpec, task files, or the repository itself?
8
+
9
+ All three must be "yes" to promote. If any is "no", the item stays in task memory.
10
+
11
+ ### Confidence signal
12
+
13
+ Tag each durable decision with a confidence in `[0.3–0.9]` plus the tasks that confirmed it — `… — [confidence: 0.7; seen in tasks 1,3,5]`:
14
+
15
+ - Confirmed in **≥2 of the last 3 tasks** with no contradiction → **≥0.7** (trust it; safe to act on without re-deriving).
16
+ - Confirmed once, or inferred but not yet reused → **0.3–0.5** (tentative; gather more signal before relying).
17
+ - Contradicted by a later task or the repo → lower it or drop the decision (see Error Handling).
18
+
19
+ Confidence makes cross-task learning explicit and is the signal `/dw-learn` reads to promote high-confidence decisions into durable instincts, constitution principles, or rules.
20
+
21
+ ### Belongs in shared memory
22
+ - A discovered constraint affecting multiple tasks ("the Stripe API rate-limits to 100 req/s — batch operations must respect this")
23
+ - A cross-cutting architectural decision made during implementation ("chose React Server Components for data fetching across the whole feature")
24
+ - An open risk future tasks must account for ("migration depends on schema v3 which is not yet deployed to staging")
25
+
26
+ ### Stays in task memory
27
+ - Files touched during this task's implementation
28
+ - Debugging steps taken to resolve a task-specific error
29
+ - The current task's objective and acceptance criteria snapshot
30
+ - A workaround applied only to the current task's scope
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: dw-minimalism
3
- description: Pre-generation YAGNI gate. Before writing new code, infra, or abstractions, climb the decision ladder (need it? reuse? stdlib? native? installed dep? one line?) at the active lite/full/ultra intensity. Triggers from /dw-run, /dw-plan, /dw-review and any "add X / build Y" intent.
3
+ description: Evaluate a proposed abstraction, module or dependency for unnecessary complexity during planning or a focused over-engineering review.
4
4
  allowed-tools:
5
5
  - Read
6
6
  - Grep
@@ -11,8 +11,7 @@ allowed-tools:
11
11
  # dw-minimalism — Write the Least Code That Solves the Problem
12
12
 
13
13
  The cheapest code to maintain is the code you never wrote. This skill runs a
14
- **pre-generation gate**: before producing a new function, file, dependency, or
15
- abstraction, justify it against a decision ladder. It is the missing rung
14
+ **pre-generation gate**: when evaluating a proposed abstraction or module, assess its necessity against the task requirements. It is the missing rung
16
15
  between `dw-search-first` (which evaluates *external dependencies*) and
17
16
  `dw-simplification` (which cleans up code *after* it exists).
18
17
 
@@ -21,9 +20,9 @@ between `dw-search-first` (which evaluates *external dependencies*) and
21
20
  Read this skill when:
22
21
 
23
22
  - `/dw-plan` is about to specify new modules/services — pressure-test scope before tasks are written.
24
- - `/dw-run` is about to implement — climb the ladder before the first line of new code.
23
+ - A task proposes an abstraction or module whose necessity is unclear.
25
24
  - `/dw-review` flags over-engineering (speculative generality, premature abstraction, unused options).
26
- - The user (or a task) says "add X", "build Y", "create a helper/wrapper/abstraction for Z".
25
+ - The user asks to assess whether a helper, wrapper, abstraction or dependency is necessary.
27
26
 
28
27
  Do NOT use to:
29
28