special-agents 0.3.0 → 0.3.2

package/content/agents/builder.yaml

@@ -19,6 +19,7 @@ conditionalPacks:
     - domain/ai-cost
     - domain/ai-reproducibility
     - domain/ai-governance
+    - domain/ai-batch-testing
   frontend:
     - domain/accessibility
 model: null

package/content/defaults.yaml

@@ -9,5 +9,5 @@ stack:
   packageManager: "pnpm"
 flags:
   hasFrontend: true
-  usesAI: false
+  usesAI: true
   runsInCloud: true

package/content/rules/domain/ai-batch-testing.md

@@ -0,0 +1,27 @@
+---
+name: ai-batch-testing
+category: domain
+description: "Batch execution and model comparison for any AI step."
+condition: ai
+template: false
+---
+## AI Batch Execution And Model Comparison
+
+- Design every AI step — generation, classification, extraction, routing, validation, summarization, or any other model-driven operation — so it can be run in batch mode without code changes.
+- Batch mode means: given a set of inputs, run all of them through the same step and collect results, scores, costs, and latency for each.
+- Build batch execution as a first-class capability, not an afterthought. The ability to run a step over many examples should be available from day one.
+- Support running a batch across multiple models simultaneously so outputs can be compared side-by-side for the same inputs.
+- For each batch run, record: input, model/provider, parameters, output, scoring result, cost, latency, total step duration, and timestamp. Store results in a format that supports filtering, sorting, and aggregation.
+- Record step timing at every level: time-to-first-token, total generation time, and end-to-end step duration including pre/post-processing. This allows latency regressions to be caught the same way correctness regressions are.
+- Track step timing trends over time — not just per run, but across runs — so gradual latency drift is visible before it becomes a user-facing problem.
+- Define a scoring function for each AI step before implementing it. Scoring may be rule-based, model-judged, human-reviewed, or a combination — but it must be explicit and repeatable.
+- Use batch comparison results as the primary evidence for model selection decisions at each step. Document the winning model, the runner-up, and the margin by which the winner was chosen.
+- Allow batch runs to use golden examples, synthetic inputs, real sampled inputs, adversarial cases, or any combination.
+- Make it easy to add new inputs to the batch suite for a step — especially real-world cases that caused problems in production.
+- Track batch suite coverage: the suite should include common cases, edge cases, known failure modes, and adversarial inputs for each step.
+- Support partial batches and incremental runs so new models or parameter changes can be tested against a subset of the suite before a full run.
+- Automate batch runs in CI when prompts, models, parameters, or scoring functions change.
+- Produce a diff-friendly summary report for each batch run: pass/fail per example, score distributions, cost and latency comparisons, and a clear recommendation.
+- When batch results show a regression, block the change and require explicit sign-off with a documented rationale before merging.
+- Keep batch suite inputs, expected outputs (or scoring criteria), and historical results versioned in the repository alongside the prompts they test.
+

package/content/rules/domain/ai-evals.md

@@ -8,18 +8,28 @@ template: false
 ## AI Quality, Confidence, And Evals
 
 - Always plan AI generation workflows for high-confidence outputs, not just plausible outputs.
-- Define quality gates, confidence scoring, QA checks, and acceptance thresholds for AI-generated results.
+- Define quality gates, confidence scoring, QA checks, and acceptance thresholds for every AI-driven step before implementing it.
 - Assume the project will include custom QA steps for important AI generations.
-- When an AI generation is important or user-facing, include tests, evals, review workflows, or human approval steps that match the risk level.
-- For prompts used in AI generation steps, QA steps, correction steps, and other relevant AI uses, instruct the model to check its work three times by default.
-- Make the number of model self-checks configurable.
-- Make extra model self-checking toggleable so it can be disabled when cost, latency, or workflow needs require it.
-- During planning, consult the user about where work-checking prompts should be used, how many checks are appropriate, and when extra checks may be turned off.
-- When using AI generation, start with at least one correct or top-quality example, and prefer many high-quality examples.
-- Use correct/high-quality examples during AI generation where they improve quality.
-- During planning, consult the user on which AI generation stages should use examples.
-- Build the ability to toggle the use of correct/high-quality examples at AI generation steps where practical.
-- Track AI failures, correction rates, confidence levels, approval outcomes, and quality trends.
+- When an AI step is important or user-facing, include tests, evals, review workflows, or human approval steps that match the risk level.
+- For prompts used in AI generation, QA, and correction steps, instruct the model to check its work three times by default. Make the number of self-checks configurable and toggleable so it can be disabled when cost, latency, or workflow requires it.
+- During planning, consult the user about where self-checking prompts should be used, how many checks are appropriate, and when extra checks may be turned off.
+
+### Correct-Example Libraries
+
+- For every AI step, build and maintain a library of correct, high-quality examples that demonstrate what a good output looks like. This library serves two purposes: it is injected into prompts as few-shot context to guide the model, and it anchors the eval suite.
+- Seed the example library with at least one correct example before writing any AI step. Prefer many high-quality, diverse examples that cover common cases, edge cases, and tricky inputs.
+- Examples should show the full input-to-output mapping for the step — not just an answer, but the reasoning or format the model is expected to follow.
+- Include negative examples (clearly wrong outputs with an explanation of why they are wrong) where they meaningfully help the model avoid common failure modes.
+- Version the example library alongside prompts. A prompt change and its associated example changes should be reviewed together.
+- During planning, identify which AI steps will use injected examples, how many to inject at runtime, and whether the selection should be static, retrieved by similarity, or chosen by another model.
+- Build the ability to toggle example injection per step so it can be disabled when cost or latency requires it.
+- Track which examples were injected for each production generation as part of the replay record so failures can be traced back to example quality.
+- Review and prune the example library over time: retire examples that no longer represent correct behavior and add examples from corrected production failures.
+
+### Eval Suites
+
 - Keep AI eval suites, golden examples, adversarial cases, and regression results updated in `docs/ai-evals.md`.
-- Promote prompt or model changes only when required eval suites meet documented thresholds, or document the explicit user-approved reason for accepting the risk.
+- Maintain eval sets that cover: correct common cases, edge cases, known failure modes, adversarial inputs, and cases drawn from corrected production failures.
+- Track AI failures, correction rates, confidence levels, approval outcomes, and quality trends.
+- Promote prompt, parameter, or model changes only when required eval suites meet documented thresholds, or document the explicit user-approved reason for accepting the risk.
 

package/content/rules/packs.index.yaml

@@ -133,45 +133,51 @@ packs:
     condition: ai
     template: false
     order: 22000
+  - name: ai-batch-testing
+    category: domain
+    path: domain/ai-batch-testing.md
+    condition: ai
+    template: false
+    order: 23000
   - name: release
     category: process
     path: process/release.md
     condition: always
     template: false
-    order: 23000
+    order: 24000
   - name: accessibility
     category: domain
     path: domain/accessibility.md
     condition: frontend
     template: false
-    order: 24000
+    order: 25000
   - name: observability
     category: domain
     path: domain/observability.md
     condition: always
     template: false
-    order: 25000
+    order: 26000
   - name: business-docs
     category: process
     path: process/business-docs.md
     condition: always
     template: false
-    order: 26000
+    order: 27000
   - name: architecture
     category: process
     path: process/architecture.md
     condition: always
     template: false
-    order: 27000
+    order: 28000
   - name: definition-of-done
     category: base
     path: base/definition-of-done.md
     condition: always
     template: false
-    order: 28000
+    order: 29000
   - name: response-expectations
     category: base
     path: base/response-expectations.md
     condition: always
     template: false
-    order: 29000
+    order: 30000

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "special-agents",
-  "version": "0.3.0",
+  "version": "0.3.2",
   "description": "Rafi library: composable best-practice rules, skills, and agents for Claude Code and Codex.",
   "type": "module",
   "main": "./dist/index.js",