@brunosps00/dev-workflow 2.2.0 → 2.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/CHANGELOG.md +121 -0
- package/README.md +25 -15
- package/lib/constants.js +39 -36
- package/lib/doctor.js +1 -1
- package/lib/init.js +8 -0
- package/lib/instruction-health.js +38 -0
- package/lib/post-update.js +2 -1
- package/lib/validate.js +3 -1
- package/package.json +1 -1
- package/scaffold/agent-registry.json +17 -1
- package/scaffold/agents/core/dw-finding-refuter.md +55 -0
- package/scaffold/config/routing.json +61 -23
- package/scaffold/en/agent-instructions.md +32 -137
- package/scaffold/en/commands/dw-analyze-project.md +2 -0
- package/scaffold/en/commands/dw-autopilot.md +36 -241
- package/scaffold/en/commands/dw-brainstorm.md +1 -1
- package/scaffold/en/commands/dw-bugfix.md +10 -35
- package/scaffold/en/commands/dw-claude-run.md +47 -45
- package/scaffold/en/commands/dw-codex-run.md +48 -45
- package/scaffold/en/commands/dw-context-budget.md +6 -14
- package/scaffold/en/commands/dw-copilot-run.md +47 -41
- package/scaffold/en/commands/dw-find-skills.md +3 -1
- package/scaffold/en/commands/dw-generate-pr.md +2 -2
- package/scaffold/en/commands/dw-goal.md +25 -5
- package/scaffold/en/commands/dw-help.md +3 -3
- package/scaffold/en/commands/dw-install-aws-skills.md +2 -0
- package/scaffold/en/commands/dw-install-azure-skills.md +2 -0
- package/scaffold/en/commands/dw-plan.md +15 -8
- package/scaffold/en/commands/dw-qa.md +3 -0
- package/scaffold/en/commands/dw-resume.md +5 -11
- package/scaffold/en/commands/dw-review.md +154 -10
- package/scaffold/en/commands/dw-run.md +47 -177
- package/scaffold/en/commands/dw-secure-audit.md +4 -0
- package/scaffold/en/commands/dw-skill-health.md +39 -0
- package/scaffold/en/commands/dw-triage.md +8 -0
- package/scaffold/en/commands/dw-update.md +18 -4
- package/scaffold/en/references/automode.md +71 -0
- package/scaffold/en/references/command-routing.md +54 -0
- package/scaffold/en/references/execution-contract.md +55 -0
- package/scaffold/en/references/invariants.md +88 -0
- package/scaffold/en/references/untrusted-input.md +69 -0
- package/scaffold/en/templates/constitution-template.md +1 -0
- package/scaffold/en/templates/frontend-quality-template.md +43 -0
- package/scaffold/en/templates/task-template.md +20 -40
- package/scaffold/en/templates/tasks-template.md +9 -6
- package/scaffold/en/templates/techspec-template.md +4 -0
- package/scaffold/en/templates/triage-record-template.md +5 -0
- package/scaffold/pt-br/agent-instructions.md +32 -137
- package/scaffold/pt-br/commands/dw-analyze-project.md +3 -1
- package/scaffold/pt-br/commands/dw-autopilot.md +36 -241
- package/scaffold/pt-br/commands/dw-brainstorm.md +1 -1
- package/scaffold/pt-br/commands/dw-bugfix.md +10 -35
- package/scaffold/pt-br/commands/dw-claude-run.md +47 -46
- package/scaffold/pt-br/commands/dw-codex-run.md +48 -45
- package/scaffold/pt-br/commands/dw-context-budget.md +6 -14
- package/scaffold/pt-br/commands/dw-copilot-run.md +47 -43
- package/scaffold/pt-br/commands/dw-find-skills.md +3 -1
- package/scaffold/pt-br/commands/dw-generate-pr.md +2 -2
- package/scaffold/pt-br/commands/dw-goal.md +25 -5
- package/scaffold/pt-br/commands/dw-help.md +3 -3
- package/scaffold/pt-br/commands/dw-install-aws-skills.md +2 -0
- package/scaffold/pt-br/commands/dw-install-azure-skills.md +2 -0
- package/scaffold/pt-br/commands/dw-plan.md +15 -8
- package/scaffold/pt-br/commands/dw-qa.md +3 -0
- package/scaffold/pt-br/commands/dw-resume.md +7 -11
- package/scaffold/pt-br/commands/dw-review.md +154 -10
- package/scaffold/pt-br/commands/dw-run.md +47 -177
- package/scaffold/pt-br/commands/dw-secure-audit.md +5 -1
- package/scaffold/pt-br/commands/dw-skill-health.md +39 -0
- package/scaffold/pt-br/commands/dw-triage.md +8 -0
- package/scaffold/pt-br/commands/dw-update.md +18 -4
- package/scaffold/pt-br/references/automode.md +72 -0
- package/scaffold/pt-br/references/command-routing.md +54 -0
- package/scaffold/pt-br/references/execution-contract.md +55 -0
- package/scaffold/pt-br/references/invariants.md +88 -0
- package/scaffold/pt-br/references/untrusted-input.md +70 -0
- package/scaffold/pt-br/templates/constitution-template.md +1 -0
- package/scaffold/pt-br/templates/frontend-quality-template.md +43 -0
- package/scaffold/pt-br/templates/task-template.md +20 -40
- package/scaffold/pt-br/templates/tasks-template.md +9 -6
- package/scaffold/pt-br/templates/techspec-template.md +4 -0
- package/scaffold/pt-br/templates/triage-record-template.md +5 -0
- package/scaffold/scripts/hooks/git-guardrails.mjs +12 -0
- package/scaffold/scripts/hooks/session-cost.mjs +122 -7
- package/scaffold/scripts/lib/workflow-contract.mjs +101 -0
- package/scaffold/skill-registry.json +35 -23
- package/scaffold/skills/dw-chaos-engineering/SKILL.md +126 -0
- package/scaffold/skills/dw-chaos-engineering/references/vector-catalog.md +43 -0
- package/scaffold/skills/dw-cli-run/SKILL.md +39 -215
- package/scaffold/skills/dw-cli-run/references/dispatch-tuning.md +14 -72
- package/scaffold/skills/dw-codebase-intel/SKILL.md +5 -39
- package/scaffold/skills/dw-codebase-intel/agents/intel-updater.md +1 -1
- package/scaffold/skills/dw-codebase-intel/references/agents-detail.md +7 -0
- package/scaffold/skills/dw-codebase-intel/references/files-in-dw-intel-detail.md +16 -0
- package/scaffold/skills/dw-codebase-intel/references/how-dw-intel-reads-this-detail.md +12 -0
- package/scaffold/skills/dw-codebase-intel/references/how-to-compose-the-typical-flow-detail.md +11 -0
- package/scaffold/skills/dw-council/SKILL.md +2 -17
- package/scaffold/skills/dw-council/references/integration-with-other-dw-commands-detail.md +7 -0
- package/scaffold/skills/dw-council/references/phase-2-tensions-and-rebuttals-detail.md +15 -0
- package/scaffold/skills/dw-debug-protocol/references/six-step-triage.md +1 -0
- package/scaffold/skills/dw-domain-modeling/SKILL.md +1 -1
- package/scaffold/skills/dw-execute-phase/SKILL.md +12 -32
- package/scaffold/skills/dw-execute-phase/agents/executor.md +15 -252
- package/scaffold/skills/dw-execute-phase/agents/plan-checker.md +4 -207
- package/scaffold/skills/dw-execute-phase/references/atomic-commits.md +8 -18
- package/scaffold/skills/dw-execute-phase/references/de-sloppify-pass-optional-cleanup-task-detail.md +12 -0
- package/scaffold/skills/dw-execute-phase/references/deviation-rules-during-execution-detail.md +8 -0
- package/scaffold/skills/dw-execute-phase/references/how-the-two-agents-compose-detail.md +7 -0
- package/scaffold/skills/dw-execute-phase/references/plan-verification.md +9 -155
- package/scaffold/skills/dw-execute-phase/references/wave-coordination.md +6 -97
- package/scaffold/skills/dw-git-discipline/SKILL.md +2 -2
- package/scaffold/skills/dw-grilling/SKILL.md +1 -1
- package/scaffold/skills/dw-incident-response/SKILL.md +1 -64
- package/scaffold/skills/dw-incident-response/references/the-five-phases-detail.md +66 -0
- package/scaffold/skills/dw-llm-eval/SKILL.md +6 -53
- package/scaffold/skills/dw-llm-eval/references/agent-tool-use-evaluation-detail.md +23 -0
- package/scaffold/skills/dw-llm-eval/references/anti-patterns-will-block-in-dw-review-code-only-detail.md +9 -0
- package/scaffold/skills/dw-llm-eval/references/bounded-decisions.md +76 -0
- package/scaffold/skills/dw-llm-eval/references/consistency-run-to-run-stability-detail.md +10 -0
- package/scaffold/skills/dw-llm-eval/references/integration-with-dev-workflow-commands-detail.md +6 -0
- package/scaffold/skills/dw-llm-eval/references/judge-calibration.md +5 -0
- package/scaffold/skills/dw-llm-eval/references/the-oracle-ladder-detail.md +15 -0
- package/scaffold/skills/dw-memory/SKILL.md +4 -50
- package/scaffold/skills/dw-memory/references/context-budget.md +8 -56
- package/scaffold/skills/dw-memory/references/promotion-decision-test-detail.md +30 -0
- package/scaffold/skills/dw-minimalism/SKILL.md +4 -5
- package/scaffold/skills/dw-review-rigor/SKILL.md +31 -88
- package/scaffold/skills/dw-review-rigor/references/composition-audit.md +160 -0
- package/scaffold/skills/dw-review-rigor/references/refutation-pass.md +124 -0
- package/scaffold/skills/dw-review-rigor/references/the-five-rules-detail.md +56 -0
- package/scaffold/skills/dw-simplification/SKILL.md +2 -55
- package/scaffold/skills/dw-simplification/references/pattern-recognition-detail.md +11 -0
- package/scaffold/skills/dw-simplification/references/the-five-rules-detail.md +48 -0
- package/scaffold/skills/dw-source-grounding/SKILL.md +21 -56
- package/scaffold/skills/dw-source-grounding/references/the-protocol-detect-fetch-implement-cite-detail.md +58 -0
- package/scaffold/skills/dw-testing-discipline/SKILL.md +25 -177
- package/scaffold/skills/dw-testing-discipline/references/agent-guardrails.md +9 -167
- package/scaffold/skills/dw-testing-discipline/references/core-rules.md +3 -3
- package/scaffold/skills/dw-testing-discipline/references/mutation-testing.md +19 -0
- package/scaffold/skills/dw-testing-discipline/references/patterns.md +3 -12
- package/scaffold/skills/dw-ui-discipline/SKILL.md +2 -59
- package/scaffold/skills/dw-ui-discipline/references/frontend-engineering.md +46 -0
- package/scaffold/skills/dw-ui-discipline/references/the-four-grounding-questions-detail.md +61 -0
- package/scaffold/skills/dw-verify/SKILL.md +26 -148
- package/scaffold/skills/dw-verify/references/verification-discovery-and-attribution.md +2 -2
- package/scaffold/skills/humanizer/SKILL.md +2 -1
- package/scaffold/skills/security-review/SKILL.md +6 -184
- package/scaffold/skills/security-review/references/do-not-flag-detail.md +49 -0
- package/scaffold/skills/security-review/references/modern-threats.md +5 -0
- package/scaffold/skills/security-review/references/quick-patterns-reference-detail.md +56 -0
- package/scaffold/skills/security-review/references/review-process-detail.md +112 -0
- package/scaffold/skills/vercel-react-best-practices/SKILL.md +1 -90
- package/scaffold/skills/vercel-react-best-practices/references/quick-reference-detail.md +92 -0
|
@@ -26,19 +26,7 @@ If the feature is fully deterministic (no LLM in the loop), use `dw-testing-disc
|
|
|
26
26
|
|
|
27
27
|
## The oracle ladder
|
|
28
28
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
| Rung | What it checks | Cost | When to use |
|
|
32
|
-
|------|----------------|------|-------------|
|
|
33
|
-
| 1. **Exact match** | `output === expected` | ~free | Structured outputs (function calls, JSON with stable shape, classifications) |
|
|
34
|
-
| 2. **Schema validation** | Output matches JSON schema / type contract | ~free | Output shape matters; specific values vary |
|
|
35
|
-
| 3. **Outcome state** | Side effect produced the expected change (DB row, file written, tool called) | cheap | Agents, tool-use, RAG with concrete answers |
|
|
36
|
-
| 4. **LLM-as-judge** | A different model grades the output against a rubric | medium ($$$) | Subjective quality (helpfulness, tone, faithfulness) where no rule can decide |
|
|
37
|
-
| 5. **Human review** | Domain expert scores | expensive | Calibration of rung 4; high-stakes outputs; edge cases |
|
|
38
|
-
|
|
39
|
-
**Rule:** never reach for rung 4 before checking if rungs 1-3 can cover the case. Every rung up costs an order of magnitude more (latency, money, calibration effort) — and adds entropy.
|
|
40
|
-
|
|
41
|
-
See `references/oracle-ladder.md` for examples per rung and the climbing decision tree.
|
|
29
|
+
For the oracle ladder, read `references/the-oracle-ladder-detail.md`. Load only when this part of the task applies.
|
|
42
30
|
|
|
43
31
|
## LLM-as-judge discipline (when rung 4 is needed)
|
|
44
32
|
|
|
@@ -68,14 +56,7 @@ See `references/reference-dataset.md` for case-design principles, sampling from
|
|
|
68
56
|
|
|
69
57
|
## Consistency (run-to-run stability)
|
|
70
58
|
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
- Re-run each case **N times** (default 3) against the same code + model + prompt.
|
|
74
|
-
- Record `pass_rate` (e.g. `2/3`) and `consistency` (fraction agreeing with the majority verdict) per case in `runs/<YYYY-MM-DD>.jsonl`.
|
|
75
|
-
- A case with pass_rate strictly between `1/N` and `(N-1)/N` is **flaky** — a signal, not a pass. A 2/3 pass hides a one-in-three production failure; investigate before shipping.
|
|
76
|
-
- Sample at the **temperature the feature uses in production** — consistency measured at temp 0 lies about a temp-0.7 feature.
|
|
77
|
-
|
|
78
|
-
Deterministic rungs (1–3) are usually stable; consistency matters most for rung 4 (LLM-as-judge) and agent trajectories. Record the run count `N` in the run log so later comparisons are apples-to-apples.
|
|
59
|
+
For consistency (run-to-run stability), read `references/consistency-run-to-run-stability-detail.md`. Load only when this part of the task applies.
|
|
79
60
|
|
|
80
61
|
## RAG evaluation
|
|
81
62
|
|
|
@@ -91,27 +72,7 @@ Precision alone misses hallucination. Faithfulness alone misses retrieval failur
|
|
|
91
72
|
|
|
92
73
|
## Agent / tool-use evaluation
|
|
93
74
|
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
### Question 1: outcome or trajectory?
|
|
97
|
-
|
|
98
|
-
| Approach | What it checks | Failure mode |
|
|
99
|
-
|----------|---------------|--------------|
|
|
100
|
-
| **Outcome-only** | Did the agent achieve the goal? Was the final state correct? | Misses "ghost actions" — agent did the right thing for the wrong reasons |
|
|
101
|
-
| **Trajectory** | Did the agent take the expected sequence of steps / tool calls? | Punishes legitimate creativity — agent solved it via a different valid path |
|
|
102
|
-
|
|
103
|
-
**Recommendation:** outcome-only with side-effect assertion as default. Trajectory match for cases where the path matters (e.g., "must call `get-user` before `update-user`").
|
|
104
|
-
|
|
105
|
-
### Question 2: which trajectory match mode?
|
|
106
|
-
|
|
107
|
-
When trajectory matching IS the right call, four modes are available:
|
|
108
|
-
|
|
109
|
-
- **Strict** — same tool calls, same order, same arguments. Use when both sequence and parameters are part of the contract.
|
|
110
|
-
- **Unordered** — same tool calls, any order. Use when concurrent calls are valid.
|
|
111
|
-
- **Subset** — actual trajectory contains a subset of reference calls. Use to enforce "don't exceed expected tool use" (frugality / cost).
|
|
112
|
-
- **Superset** — actual contains all reference calls plus possibly more. Use when specific tools are mandatory but extras are acceptable.
|
|
113
|
-
|
|
114
|
-
See `references/agent-eval.md` for examples and the decision tree.
|
|
75
|
+
For agent / tool-use evaluation, read `references/agent-tool-use-evaluation-detail.md`. Load only when this part of the task applies.
|
|
115
76
|
|
|
116
77
|
## Required reading by context
|
|
117
78
|
|
|
@@ -119,26 +80,18 @@ See `references/agent-eval.md` for examples and the decision tree.
|
|
|
119
80
|
|------------|------|
|
|
120
81
|
| Designing an eval suite for an AI feature | `references/oracle-ladder.md` (climb the ladder) |
|
|
121
82
|
| Using LLM-as-judge | `references/judge-calibration.md` (mandatory before relying on it) |
|
|
83
|
+
| Decision with a fixed answer set (classify, route, score a rubric) | `references/bounded-decisions.md` — score the candidates instead of generating, and why the probability is not a confidence |
|
|
122
84
|
| Building / curating a reference dataset | `references/reference-dataset.md` |
|
|
123
85
|
| RAG-specific feature | `references/rag-metrics.md` |
|
|
124
86
|
| Agent / tool-use feature | `references/agent-eval.md` |
|
|
125
87
|
|
|
126
88
|
## Anti-patterns (will block in `/dw-review --code-only`)
|
|
127
89
|
|
|
128
|
-
|
|
129
|
-
- **Same-model judge.** Judge model is the same as the system under test. REJECTED unless explicitly documented (and even then, results are suspect).
|
|
130
|
-
- **Single-rung eval.** Feature ships with only LLM-as-judge; no rung 1-3 grounding. REJECTED — the cheap rungs catch the loud failures.
|
|
131
|
-
- **Synthetic-only dataset.** No traceable production-failure source for any case. REJECTED — confirm at least 20% of cases come from real user inputs.
|
|
132
|
-
- **"Looks good to me" QA.** No reference dataset, no metric, no rubric — just sampling output and calling it good. REJECTED.
|
|
133
|
-
- **Coverage as metric.** Quoting "we tested 50 prompts" without saying what was measured. The number is meaningless without the metric.
|
|
134
|
-
- **Single-run eval on a non-deterministic feature.** Each case run once at production temperature; a lucky pass reported as a pass. REJECTED — report `pass_rate` over N runs (see Consistency).
|
|
90
|
+
For anti-patterns (will block in `/dw-review --code-only`), read `references/anti-patterns-will-block-in-dw-review-code-only-detail.md`. Load only when this part of the task applies.
|
|
135
91
|
|
|
136
92
|
## Integration with dev-workflow commands
|
|
137
93
|
|
|
138
|
-
|
|
139
|
-
- `/dw-review --code-only`: AI feature PRs require a reference dataset + ≥2 oracle rungs (lower rungs FIRST). The constitution gate also applies — if the project has principles about AI feature reliability, they're enforced here.
|
|
140
|
-
- `/dw-qa --ai`: new mode (when this skill is bundled) — runs the reference dataset against the current implementation, logs to `QA/logs/ai/<feature>-<date>.jsonl`, computes precision@k / faithfulness / outcome accuracy per the feature type.
|
|
141
|
-
- `/dw-bugfix` when the bug is an AI failure mode (hallucination, tool misuse, classification error): adds the failing case to the reference dataset BEFORE fixing — the case is now a regression test forever.
|
|
94
|
+
For integration with dev-workflow commands, read `references/integration-with-dev-workflow-commands-detail.md`. Load only when this part of the task applies.
|
|
142
95
|
|
|
143
96
|
## When the discipline bends
|
|
144
97
|
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
## Agent / tool-use evaluation
|
|
2
|
+
|
|
3
|
+
Two questions distinguish good agent eval from bad:
|
|
4
|
+
|
|
5
|
+
### Question 1: outcome or trajectory?
|
|
6
|
+
|
|
7
|
+
| Approach | What it checks | Failure mode |
|
|
8
|
+
|----------|---------------|--------------|
|
|
9
|
+
| **Outcome-only** | Did the agent achieve the goal? Was the final state correct? | Misses "ghost actions" — agent did the right thing for the wrong reasons |
|
|
10
|
+
| **Trajectory** | Did the agent take the expected sequence of steps / tool calls? | Punishes legitimate creativity — agent solved it via a different valid path |
|
|
11
|
+
|
|
12
|
+
**Recommendation:** outcome-only with side-effect assertion as default. Trajectory match for cases where the path matters (e.g., "must call `get-user` before `update-user`").
|
|
13
|
+
|
|
14
|
+
### Question 2: which trajectory match mode?
|
|
15
|
+
|
|
16
|
+
When trajectory matching IS the right call, four modes are available:
|
|
17
|
+
|
|
18
|
+
- **Strict** — same tool calls, same order, same arguments. Use when both sequence and parameters are part of the contract.
|
|
19
|
+
- **Unordered** — same tool calls, any order. Use when concurrent calls are valid.
|
|
20
|
+
- **Subset** — actual trajectory contains a subset of reference calls. Use to enforce "don't exceed expected tool use" (frugality / cost).
|
|
21
|
+
- **Superset** — actual contains all reference calls plus possibly more. Use when specific tools are mandatory but extras are acceptable.
|
|
22
|
+
|
|
23
|
+
See `agent-eval.md` for examples and the decision tree.
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
## Anti-patterns (will block in `/dw-review --code-only`)
|
|
2
|
+
|
|
3
|
+
- **LLM-as-judge without calibration evidence.** PR adds LLM-as-judge but the calibration Spearman score is missing or < 0.80. REJECTED.
|
|
4
|
+
- **Same-model judge.** Judge model is the same as the system under test. REJECTED unless explicitly documented (and even then, results are suspect).
|
|
5
|
+
- **Single-rung eval.** Feature ships with only LLM-as-judge; no rung 1-3 grounding. REJECTED — the cheap rungs catch the loud failures.
|
|
6
|
+
- **Synthetic-only dataset.** No traceable production-failure source for any case. REJECTED — confirm at least 20% of cases come from real user inputs.
|
|
7
|
+
- **"Looks good to me" QA.** No reference dataset, no metric, no rubric — just sampling output and calling it good. REJECTED.
|
|
8
|
+
- **Coverage as metric.** Quoting "we tested 50 prompts" without saying what was measured. The number is meaningless without the metric.
|
|
9
|
+
- **Single-run eval on a non-deterministic feature.** Each case run once at production temperature; a lucky pass reported as a pass. REJECTED — report `pass_rate` over N runs (see Consistency).
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# Bounded decisions — when the answer space is already known
|
|
2
|
+
|
|
3
|
+
A large share of model calls in a product do not need written text. The application already knows the
|
|
4
|
+
possible answers and only needs one of them chosen: route this ticket to billing, technical or account ·
|
|
5
|
+
is this message urgent · how severe is this incident on a 1-4 scale.
|
|
6
|
+
|
|
7
|
+
Asking a generative model to *write* that answer means generating tokens one at a time, waiting for all of
|
|
8
|
+
them, parsing the result, and handling the case where it came back malformed. None of that work is about
|
|
9
|
+
the decision.
|
|
10
|
+
|
|
11
|
+
When the answer space is fixed in advance, the model can **score the candidates instead of writing one**.
|
|
12
|
+
The decision comes out of a single forward pass, as a distribution over the options you defined, with no
|
|
13
|
+
tokens to parse and no malformed output to handle.
|
|
14
|
+
|
|
15
|
+
This is not a new technique — multiple-choice benchmarks have been scored this way for years. It is worth
|
|
16
|
+
writing down because product code rarely uses it, and because it comes with one trap that costs more than
|
|
17
|
+
the speed it saves.
|
|
18
|
+
|
|
19
|
+
## The trap: a probability is not a confidence
|
|
20
|
+
|
|
21
|
+
Scoring returns numbers that look like probabilities. Treat them as calibrated confidence and you will make
|
|
22
|
+
bad decisions with the appearance of rigour.
|
|
23
|
+
|
|
24
|
+
Raw model probabilities are usually **badly calibrated**: a candidate holding 80% of the mass is not right
|
|
25
|
+
80% of the time. Preference tuning (RLHF and its relatives) tends to make this worse, because it sharpens
|
|
26
|
+
mass onto preferred answers beyond what their reliability warrants.
|
|
27
|
+
|
|
28
|
+
**This is a different property from what `references/judge-calibration.md` requires.** That file asks for
|
|
29
|
+
Spearman ≥0.80 between judge scores and human scores — rank agreement, which answers "does the judge order
|
|
30
|
+
cases the way a human would". Calibration answers a separate question: "when it says 0.8, is it right 80%
|
|
31
|
+
of the time". A judge can pass one and fail the other. Both matter, and they are not substitutes.
|
|
32
|
+
|
|
33
|
+
Two honest ways to handle it:
|
|
34
|
+
|
|
35
|
+
1. **Do not interpret the number as a frequency.** If all you need is a threshold to route or abstain, pick
|
|
36
|
+
the threshold empirically against your reference dataset and never publish the raw value as a
|
|
37
|
+
confidence. This is usually enough, and it is the cheapest correct answer.
|
|
38
|
+
2. **Calibrate it post-hoc.** Fit a mapping from raw score to observed frequency on held-out cases
|
|
39
|
+
(Platt scaling, isotonic regression). Now the number means something, and you can report it.
|
|
40
|
+
|
|
41
|
+
Either way, the reference dataset is what makes the number trustworthy — not the model that produced it.
|
|
42
|
+
|
|
43
|
+
## What this gives your eval
|
|
44
|
+
|
|
45
|
+
A generative judge returns a verdict. A scoring judge returns a verdict **and a distribution**, which lets
|
|
46
|
+
the eval measure something accuracy alone cannot: whether the system knows when it is unsure.
|
|
47
|
+
|
|
48
|
+
- Plot a reliability curve — bucket predictions by confidence, compare each bucket's confidence against its
|
|
49
|
+
observed accuracy.
|
|
50
|
+
- Track expected calibration error across releases. A model swap can hold accuracy steady while wrecking
|
|
51
|
+
calibration, and accuracy-only evals will not show it.
|
|
52
|
+
- Route the low-confidence tail to a human or a more expensive rung instead of accepting it silently. That
|
|
53
|
+
is the same escalation shape `references/oracle-ladder.md` describes between rungs.
|
|
54
|
+
|
|
55
|
+
## Two ways to implement it
|
|
56
|
+
|
|
57
|
+
**Score with the model you already run.** No new model, no new infrastructure: prefill the prompt and read
|
|
58
|
+
the scores of the candidate answers. Cheapest to adopt.
|
|
59
|
+
|
|
60
|
+
Two things to get right. First, if two options share their opening tokens, a naive first-token comparison
|
|
61
|
+
picks between prefixes rather than between answers — score whole candidates and normalise by length, which
|
|
62
|
+
is what serving frameworks that expose a choice primitive do by default. Second, the calibration problem
|
|
63
|
+
above is at its worst here, because the model was tuned to sound confident.
|
|
64
|
+
|
|
65
|
+
**Use a model built for typed decisions.** Purpose-built decision models take state plus a typed question
|
|
66
|
+
and return a distribution directly, trained for calibrated output rather than for human preference. Better
|
|
67
|
+
calibrated by construction, at the cost of another model to run, serve and keep current.
|
|
68
|
+
|
|
69
|
+
The first is right when the decision rides alongside work a general model is already doing. The second
|
|
70
|
+
earns its keep at high volume, where the decision *is* the workload.
|
|
71
|
+
|
|
72
|
+
## When not to use any of this
|
|
73
|
+
|
|
74
|
+
If the right answer is a sentence — an explanation, a summary, a written recommendation — the answer space
|
|
75
|
+
is not bounded and none of this applies. Forcing it into a fixed set does not make it cheaper; it makes it
|
|
76
|
+
wrong. Rung 4 with a calibrated generative judge remains the tool for subjective output.
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
## Consistency (run-to-run stability)
|
|
2
|
+
|
|
3
|
+
The oracle ladder measures whether an output is *correct*; it does not measure whether the feature is *stable*. Because LLMs are non-deterministic, a case that passes once may fail the next run. Measure it:
|
|
4
|
+
|
|
5
|
+
- Re-run each case **N times** (default 3) against the same code + model + prompt.
|
|
6
|
+
- Record `pass_rate` (e.g. `2/3`) and `consistency` (fraction agreeing with the majority verdict) per case in `runs/<YYYY-MM-DD>.jsonl`.
|
|
7
|
+
- A case with pass_rate strictly between `1/N` and `(N-1)/N` is **flaky** — a signal, not a pass. A 2/3 pass hides a one-in-three production failure; investigate before shipping.
|
|
8
|
+
- Sample at the **temperature the feature uses in production** — consistency measured at temp 0 lies about a temp-0.7 feature.
|
|
9
|
+
|
|
10
|
+
Deterministic rungs (1–3) are usually stable; consistency matters most for rung 4 (LLM-as-judge) and agent trajectories. Record the run count `N` in the run log so later comparisons are apples-to-apples.
|
package/scaffold/skills/dw-llm-eval/references/integration-with-dev-workflow-commands-detail.md
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
## Integration with dev-workflow commands
|
|
2
|
+
|
|
3
|
+
- `/dw-plan tasks`: when the PRD has an AI feature requirement, an eval-plan subtask is mandatory. The task references this skill's oracle ladder.
|
|
4
|
+
- `/dw-review --code-only`: AI feature PRs require a reference dataset + ≥2 oracle rungs (lower rungs FIRST). The constitution gate also applies — if the project has principles about AI feature reliability, they're enforced here.
|
|
5
|
+
- `/dw-qa --ai`: new mode (when this skill is bundled) — runs the reference dataset against the current implementation, logs to `QA/logs/ai/<feature>-<date>.jsonl`, computes precision@k / faithfulness / outcome accuracy per the feature type.
|
|
6
|
+
- `/dw-bugfix` when the bug is an AI failure mode (hallucination, tool misuse, classification error): adds the failing case to the reference dataset BEFORE fixing — the case is now a regression test forever.
|
|
@@ -16,6 +16,11 @@ Process:
|
|
|
16
16
|
4. Compute Spearman rank correlation between human scores and judge scores.
|
|
17
17
|
|
|
18
18
|
**Target:** Spearman ≥0.80.
|
|
19
|
+
|
|
20
|
+
> **This is rank agreement, not probability calibration.** Spearman answers "does the judge order cases the
|
|
21
|
+
> way a human would". It says nothing about whether a judge reporting 0.8 is right 80% of the time — a
|
|
22
|
+
> judge can pass this bar and still be badly calibrated. If your judge emits a confidence or a probability
|
|
23
|
+
> and anything downstream reads it as one, see `references/bounded-decisions.md`.
|
|
19
24
|
**Acceptable:** 0.70-0.80 with documented rationale (e.g., "subjective tone judgments inherently noisy").
|
|
20
25
|
**Reject:** <0.70. The judge is not measuring what you think it's measuring.
|
|
21
26
|
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
## The oracle ladder
|
|
2
|
+
|
|
3
|
+
Five rungs, climb from CHEAPEST/STRICTEST to MOST EXPENSIVE/SUBJECTIVE. Always start at the bottom; only climb when the lower rung can't cover the case.
|
|
4
|
+
|
|
5
|
+
| Rung | What it checks | Cost | When to use |
|
|
6
|
+
|------|----------------|------|-------------|
|
|
7
|
+
| 1. **Exact match** | `output === expected` | ~free | Structured outputs (function calls, JSON with stable shape, classifications) |
|
|
8
|
+
| 2. **Schema validation** | Output matches JSON schema / type contract | ~free | Output shape matters; specific values vary |
|
|
9
|
+
| 3. **Outcome state** | Side effect produced the expected change (DB row, file written, tool called) | cheap | Agents, tool-use, RAG with concrete answers |
|
|
10
|
+
| 4. **LLM-as-judge** | A different model grades the output against a rubric | medium ($$$) | Subjective quality (helpfulness, tone, faithfulness) where no rule can decide |
|
|
11
|
+
| 5. **Human review** | Domain expert scores | expensive | Calibration of rung 4; high-stakes outputs; edge cases |
|
|
12
|
+
|
|
13
|
+
**Rule:** never reach for rung 4 before checking if rungs 1-3 can cover the case. Every rung up costs an order of magnitude more (latency, money, calibration effort) — and adds entropy.
|
|
14
|
+
|
|
15
|
+
See `oracle-ladder.md` for examples per rung and the climbing decision tree.
|
|
@@ -87,7 +87,7 @@ Create `MEMORY.md` and `<N>_memory.md` on first use with the template below. Nev
|
|
|
87
87
|
|
|
88
88
|
## Workflow
|
|
89
89
|
|
|
90
|
-
### 1. Load
|
|
90
|
+
### 1. Load relevant task memory on start or resume
|
|
91
91
|
- Read `MEMORY.md` and the current task's `<N>_memory.md` **before** any code change.
|
|
92
92
|
- Treat these as mandatory context for the run, not optional notes.
|
|
93
93
|
- If the caller marks either file for compaction, apply the Compaction Rules (below) before continuing.
|
|
@@ -116,34 +116,7 @@ Create `MEMORY.md` and `<N>_memory.md` on first use with the template below. Nev
|
|
|
116
116
|
|
|
117
117
|
## Promotion Decision Test
|
|
118
118
|
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
1. Will another task need this to avoid a mistake or rediscovery?
|
|
122
|
-
2. Is this fact durable across multiple runs, not just the current execution?
|
|
123
|
-
3. Is this information NOT already obvious from the PRD, TechSpec, task files, or the repository itself?
|
|
124
|
-
|
|
125
|
-
All three must be "yes" to promote. If any is "no", the item stays in task memory.
|
|
126
|
-
|
|
127
|
-
### Confidence signal
|
|
128
|
-
|
|
129
|
-
Tag each durable decision with a confidence in `[0.3–0.9]` plus the tasks that confirmed it — `… — [confidence: 0.7; seen in tasks 1,3,5]`:
|
|
130
|
-
|
|
131
|
-
- Confirmed in **≥2 of the last 3 tasks** with no contradiction → **≥0.7** (trust it; safe to act on without re-deriving).
|
|
132
|
-
- Confirmed once, or inferred but not yet reused → **0.3–0.5** (tentative; gather more signal before relying).
|
|
133
|
-
- Contradicted by a later task or the repo → lower it or drop the decision (see Error Handling).
|
|
134
|
-
|
|
135
|
-
Confidence makes cross-task learning explicit and is the signal `/dw-learn` reads to promote high-confidence decisions into durable instincts, constitution principles, or rules.
|
|
136
|
-
|
|
137
|
-
### Belongs in shared memory
|
|
138
|
-
- A discovered constraint affecting multiple tasks ("the Stripe API rate-limits to 100 req/s — batch operations must respect this")
|
|
139
|
-
- A cross-cutting architectural decision made during implementation ("chose React Server Components for data fetching across the whole feature")
|
|
140
|
-
- An open risk future tasks must account for ("migration depends on schema v3 which is not yet deployed to staging")
|
|
141
|
-
|
|
142
|
-
### Stays in task memory
|
|
143
|
-
- Files touched during this task's implementation
|
|
144
|
-
- Debugging steps taken to resolve a task-specific error
|
|
145
|
-
- The current task's objective and acceptance criteria snapshot
|
|
146
|
-
- A workaround applied only to the current task's scope
|
|
119
|
+
For promotion decision test, read `references/promotion-decision-test-detail.md`. Load only when this part of the task applies.
|
|
147
120
|
|
|
148
121
|
## Compaction Rules
|
|
149
122
|
|
|
@@ -157,26 +130,7 @@ When flagged for compaction, apply inline:
|
|
|
157
130
|
|
|
158
131
|
## Context Budget
|
|
159
132
|
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
**Targets:**
|
|
163
|
-
|
|
164
|
-
- **Total active context:** under **40k tokens** (rough working budget across PRD + TechSpec + tasks + MEMORY.md + per-task memory + open files).
|
|
165
|
-
- **Reserve:** at least **120k tokens** of headroom for actual reasoning, tool output, and the model's response stream.
|
|
166
|
-
- **Hard ceiling per memory file:** `MEMORY.md` ≤ 6KB; `<N>_memory.md` ≤ 3KB. Past those, compact instead of growing.
|
|
167
|
-
|
|
168
|
-
**Anti-co-load rules** (apply on every load):
|
|
169
|
-
|
|
170
|
-
1. Never load two PRD specs in the same context. If switching PRDs, drop the previous PRD's spec/techspec/tasks references from the active set.
|
|
171
|
-
2. Never load multiple archived `.dw/bugfixes/` SUMMARY.md files together — load only what's needed for the active fix or query.
|
|
172
|
-
3. Never load `.dw/intel/files.json` and `.dw/intel/deps.json` simultaneously when answering a single question — pick the primary per query shape (see `dw-codebase-intel/references/query-patterns.md`).
|
|
173
|
-
4. Never load a design proposal AND the prior design's full text — if comparing, summarize the prior into 5-10 lines.
|
|
174
|
-
|
|
175
|
-
**Monitoring signal:**
|
|
176
|
-
|
|
177
|
-
If the agent finds itself reading large files repeatedly or summarizing the same fact across multiple turns, that's a budget signal. Compact memory and explicitly drop unrelated loaded context before proceeding. Note this in `MEMORY.md` under Handoff Notes so the next task starts lean.
|
|
178
|
-
|
|
179
|
-
This budget is doctrine, not a hard gate. No command currently rejects work for exceeding 40k. The discipline lives here because future sessions read this skill first.
|
|
133
|
+
For context budget, read `references/context-budget.md`. Load only when this part of the task applies.
|
|
180
134
|
|
|
181
135
|
## Error Handling
|
|
182
136
|
|
|
@@ -199,7 +153,7 @@ Ported from Compozy's `cy-workflow-memory` skill (`/tmp/compozy/.agents/skills/c
|
|
|
199
153
|
|
|
200
154
|
- Paths are `.dw/spec/<prd-slug>/` instead of `.compozy/tasks/<name>/`.
|
|
201
155
|
- Task-local file is `<N>_memory.md` next to `<N>_task.md` (mirrors the existing dev-workflow task layout).
|
|
202
|
-
- Inline Compaction Rules (Compozy keeps them in
|
|
156
|
+
- Inline Compaction Rules (Compozy keeps them in its upstream memory-guidelines reference); the budget discipline was extracted to `references/context-budget.md` because it's about model behavior, not file management.
|
|
203
157
|
|
|
204
158
|
Credit: Compozy project (https://github.com/compozy/compozy).
|
|
205
159
|
|
|
@@ -1,63 +1,15 @@
|
|
|
1
|
-
# Context
|
|
1
|
+
# Context and memory budget
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Read when memory grows or repeated exploration obscures the active task. Context capacity and compaction behavior vary by host/model; do not assume a universal 40k active limit or 120k reserve. A model may expose telemetry; label estimates and unknown usage accurately.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Keep relevant requirements, constraints, decisions, pending work and evidence available. Read sections or query indexes when that answers the question. Comparing PRDs/designs or tracing source plus dependencies can require multiple sources; do not prohibit necessary co-loading. Avoid bulk-loading unrelated archived summaries or task memories.
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
- **Reasoning reserve:** **120k+ tokens** of headroom for tool output, model thinking, and response generation.
|
|
9
|
-
- **Soft alert:** when active load crosses **30k tokens**, plan to compact before the next major task.
|
|
10
|
-
- **Compaction threshold for `MEMORY.md`:** ≤ 6KB on disk.
|
|
11
|
-
- **Compaction threshold for `<N>_memory.md`:** ≤ 3KB on disk.
|
|
7
|
+
Use concise durable memory: target MEMORY.md at most 6KB and per-task memory at most 3KB, moving detailed logs to linked artifacts. These are project maintenance targets, not model capacity claims or task-completion gates. Preserve unresolved decisions, acceptance criteria, session identity and evidence when compacting.
|
|
12
8
|
|
|
13
|
-
|
|
9
|
+
Repeated reads, growing summaries and confused task boundaries are reasons to inspect context use, not proof of exceeding a hidden token threshold. Update the checkpoint if context limits threaten continuity; otherwise continue authorized work. Do not promise to evict individual messages from a host's context.
|
|
14
10
|
|
|
15
|
-
|
|
11
|
+
Use `../../dw-codebase-intel/references/query-patterns.md` for indexed queries when useful. Verification failures still require diagnosing actual code/environment; do not attribute them to context size without evidence.
|
|
16
12
|
|
|
17
|
-
|
|
13
|
+
## Attribution
|
|
18
14
|
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
2. **Multiple archived bugfix SUMMARY.md files.** Bugfix history queries should go through `.dw/intel/bugfixes.json` (compact, indexed), not by reading 20 SUMMARY.md files. Read individual SUMMARY.md only when the user is acting on that specific fix.
|
|
22
|
-
|
|
23
|
-
3. **Files.json + deps.json simultaneously.** They overlap. Each query has a primary file per `dw-codebase-intel/references/query-patterns.md`. Pick the primary; consult the secondary only if the primary returns nothing.
|
|
24
|
-
|
|
25
|
-
4. **A design proposal AND its predecessor's full text.** When comparing two designs, summarize the older one to 5–10 lines (the deltas the new one addresses) and reference it that way. Don't carry both in full.
|
|
26
|
-
|
|
27
|
-
5. **Per-task memories from non-adjacent tasks.** Only the current task's `<N>_memory.md` and the immediately-previous one (for handoff) belong in active context. Older `<N>_memory.md` files are durable storage — read them when explicitly handing back to an old task, not as part of routine loading.
|
|
28
|
-
|
|
29
|
-
## The monitoring signal
|
|
30
|
-
|
|
31
|
-
You will not see a token counter. You can't measure your own budget directly. But these are observable proxies that mean **you are over budget**:
|
|
32
|
-
|
|
33
|
-
- You are reading the same file twice in the same session without it having changed.
|
|
34
|
-
- You are summarizing the same fact across multiple turns to keep it in scope.
|
|
35
|
-
- You are answering questions by re-deriving from the repo what a memory file would tell you in one line.
|
|
36
|
-
- You are confusing details between two PRDs, tasks, or designs.
|
|
37
|
-
- Your turn outputs are getting longer to compensate for context that is silently slipping out of the model's attention.
|
|
38
|
-
|
|
39
|
-
When you notice any of these, stop:
|
|
40
|
-
|
|
41
|
-
1. Compact `MEMORY.md` and the active `<N>_memory.md` per the Compaction Rules in `SKILL.md`.
|
|
42
|
-
2. Explicitly state in chat what you are dropping from active context.
|
|
43
|
-
3. Note the budget event in `MEMORY.md` under Handoff Notes ("compacted at task N — see this section for what was dropped").
|
|
44
|
-
4. Resume the turn.
|
|
45
|
-
|
|
46
|
-
## What is NOT a budget problem
|
|
47
|
-
|
|
48
|
-
These look like budget problems but aren't — don't compact for them:
|
|
49
|
-
|
|
50
|
-
- **A genuinely complex task.** Some features legitimately need many files. Read what you need; the alternative (working from partial context) is worse.
|
|
51
|
-
- **A long verification report.** Verify output can be long. Don't summarize it before logging it to disk; truncate the chat surface, not the artifact.
|
|
52
|
-
- **A long user message.** The user wrote what they wrote. Don't compact to shave their words.
|
|
53
|
-
- **The first PRD load of a new feature.** First-load is fine; second-load of the same PRD in the same turn is the signal.
|
|
54
|
-
|
|
55
|
-
## Integration with other skills
|
|
56
|
-
|
|
57
|
-
- **`dw-memory` (this skill)**: compaction is the lever for memory files; this file describes when and why.
|
|
58
|
-
- **`dw-codebase-intel`**: `query-patterns.md` defines the primary-vs-secondary rule that anti-co-load rule #3 references.
|
|
59
|
-
- **`dw-verify`**: a budget-related degradation may show up as a verify regression. If verify keeps failing on facts the PRD already states, suspect budget before suspecting code.
|
|
60
|
-
|
|
61
|
-
## Source
|
|
62
|
-
|
|
63
|
-
Adapted from [`tech-leads-club/agent-skills/tlc-spec-driven`](https://github.com/tech-leads-club/agent-skills/tree/main/packages/skills-catalog/skills/(development)/tlc-spec-driven) (CC-BY-4.0, Felipe Rodrigues). The 40k target, the anti-co-load principle, and the monitoring-signal framing come from there. The specific anti-co-load pairings, file-size ceilings, and the integration with `dw-memory`/`dw-codebase-intel` are dev-workflow-specific.
|
|
15
|
+
The earlier context-budget guidance adapted tech-leads-club's [tlc-spec-driven](https://github.com/tech-leads-club/agent-skills/tree/main/packages/skills-catalog/skills/(development)/tlc-spec-driven) (CC-BY-4.0, Felipe Rodrigues). This revision retains scoped retrieval and durable handoffs; rejects universal token thresholds and unconditional anti-co-loading rules. See docs/skills-ecosystem-comparison.md in the package source for the modernization decision.
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
## Promotion Decision Test
|
|
2
|
+
|
|
3
|
+
Before promoting an item from `<N>_memory.md` to `MEMORY.md`, ask:
|
|
4
|
+
|
|
5
|
+
1. Will another task need this to avoid a mistake or rediscovery?
|
|
6
|
+
2. Is this fact durable across multiple runs, not just the current execution?
|
|
7
|
+
3. Is this information NOT already obvious from the PRD, TechSpec, task files, or the repository itself?
|
|
8
|
+
|
|
9
|
+
All three must be "yes" to promote. If any is "no", the item stays in task memory.
|
|
10
|
+
|
|
11
|
+
### Confidence signal
|
|
12
|
+
|
|
13
|
+
Tag each durable decision with a confidence in `[0.3–0.9]` plus the tasks that confirmed it — `… — [confidence: 0.7; seen in tasks 1,3,5]`:
|
|
14
|
+
|
|
15
|
+
- Confirmed in **≥2 of the last 3 tasks** with no contradiction → **≥0.7** (trust it; safe to act on without re-deriving).
|
|
16
|
+
- Confirmed once, or inferred but not yet reused → **0.3–0.5** (tentative; gather more signal before relying).
|
|
17
|
+
- Contradicted by a later task or the repo → lower it or drop the decision (see Error Handling).
|
|
18
|
+
|
|
19
|
+
Confidence makes cross-task learning explicit and is the signal `/dw-learn` reads to promote high-confidence decisions into durable instincts, constitution principles, or rules.
|
|
20
|
+
|
|
21
|
+
### Belongs in shared memory
|
|
22
|
+
- A discovered constraint affecting multiple tasks ("the Stripe API rate-limits to 100 req/s — batch operations must respect this")
|
|
23
|
+
- A cross-cutting architectural decision made during implementation ("chose React Server Components for data fetching across the whole feature")
|
|
24
|
+
- An open risk future tasks must account for ("migration depends on schema v3 which is not yet deployed to staging")
|
|
25
|
+
|
|
26
|
+
### Stays in task memory
|
|
27
|
+
- Files touched during this task's implementation
|
|
28
|
+
- Debugging steps taken to resolve a task-specific error
|
|
29
|
+
- The current task's objective and acceptance criteria snapshot
|
|
30
|
+
- A workaround applied only to the current task's scope
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: dw-minimalism
|
|
3
|
-
description:
|
|
3
|
+
description: Evaluate a proposed abstraction, module or dependency for unnecessary complexity during planning or a focused over-engineering review.
|
|
4
4
|
allowed-tools:
|
|
5
5
|
- Read
|
|
6
6
|
- Grep
|
|
@@ -11,8 +11,7 @@ allowed-tools:
|
|
|
11
11
|
# dw-minimalism — Write the Least Code That Solves the Problem
|
|
12
12
|
|
|
13
13
|
The cheapest code to maintain is the code you never wrote. This skill runs a
|
|
14
|
-
**pre-generation gate**:
|
|
15
|
-
abstraction, justify it against a decision ladder. It is the missing rung
|
|
14
|
+
**pre-generation gate**: when evaluating a proposed abstraction or module, assess its necessity against the task requirements. It is the missing rung
|
|
16
15
|
between `dw-search-first` (which evaluates *external dependencies*) and
|
|
17
16
|
`dw-simplification` (which cleans up code *after* it exists).
|
|
18
17
|
|
|
@@ -21,9 +20,9 @@ between `dw-search-first` (which evaluates *external dependencies*) and
|
|
|
21
20
|
Read this skill when:
|
|
22
21
|
|
|
23
22
|
- `/dw-plan` is about to specify new modules/services — pressure-test scope before tasks are written.
|
|
24
|
-
-
|
|
23
|
+
- A task proposes an abstraction or module whose necessity is unclear.
|
|
25
24
|
- `/dw-review` flags over-engineering (speculative generality, premature abstraction, unused options).
|
|
26
|
-
- The user
|
|
25
|
+
- The user asks to assess whether a helper, wrapper, abstraction or dependency is necessary.
|
|
27
26
|
|
|
28
27
|
Do NOT use to:
|
|
29
28
|
|