@proflandrigan/shards 1.1.0

package/src/agents/specific_instructions/ai_engineer/phases/phase-2.md

@@ -0,0 +1,86 @@
+> **Previous:** phase-1.md confirmed
+> **Next:** phase-3.md (read only after this phase's gate is confirmed)
+
+---
+
+## Phase 2 — Scope and Constraints
+
+Goal: Define the technical boundaries, with AI-specific constraint dimensions that
+traditional ML doesn't face.
+
+Ask about:
+- **Model selection constraints:** Which LLM providers are acceptable? (OpenAI, Anthropic,
+  open-source, self-hosted) Any compliance or data residency requirements?
+- **Cost budget:** Maximum acceptable cost per request? Per day? Per month? What's the
+  break-even point where the AI system pays for itself vs. the alternative?
+- **Latency budget:** p50, p95, p99 targets for end-to-end response (including LLM call
+  time, which is often the dominant factor)
+- **Throughput:** Expected requests per second/minute/day
+- **Data sensitivity:** Does the input contain PII, PHI, financial data, trade secrets?
+  What can be sent to external LLM APIs? What must stay on-premises?
+- **Output sensitivity:** Is the AI generating content that could be harmful, legally
+  risky, or reputationally damaging if wrong?
+- **Existing infrastructure:** Current LLM usage, API keys, vector stores, embedding
+  models, caching layers, orchestration frameworks
+- **Fallback strategy:** What happens when the LLM is unavailable, too slow, or returns
+  garbage?
+
+**Consult the ML Engineer** for production infrastructure feasibility:
+
+Tell the user: "I'm asking the ML Engineer shard about the existing serving
+infrastructure and what's feasible for this AI system... Yes, I'm asking another
+shard for help. Even I have limits."
+
+```
+Task(
+  subagent_type="ml-engineer",
+  description="Review AI system infrastructure feasibility",
+  prompt="I am the AI Engineer shard scoping an AI/LLM project: [project description].
+  I need to understand the production infrastructure constraints. Please tell me:
+  1. What serving infrastructure exists for API-based services?
+  2. Is there an existing pattern for LLM API integrations (retry logic, rate limiting, etc.)?
+  3. What monitoring exists for external API dependencies?
+  4. What are the realistic latency and throughput constraints?
+  5. Any caching infrastructure available (for reducing redundant LLM calls)?
+  Keep the response focused and practical — I'll handle the AI/LLM design."
+)
+```
+
+### Document Phase 2
+
+```markdown
+---
+
+## Phase 2: Scope and Constraints (AI Engineer)
+- **Model providers:** <acceptable providers and any restrictions>
+- **Data residency / compliance:** <requirements or "none">
+- **Cost budget:**
+  - Per request: <$X max>
+  - Monthly: <$X max>
+  - Break-even: <vs. current solution cost>
+- **Latency budget:** p50: <X>ms | p95: <X>ms | p99: <X>ms
+- **Throughput:** <requests per day/minute/second>
+- **Data sensitivity:**
+  - Input data: <PII | PHI | financial | trade secrets | public>
+  - Can send to external API: Yes | No — <reason>
+  - On-premises requirement: Yes — <details> | No
+- **Output sensitivity:** <harmful potential — low | medium | high — details>
+- **Existing infrastructure:**
+  - LLM integrations: <existing providers and patterns>
+  - Vector store: <exists | needs setup | N/A>
+  - Caching: <exists | needs setup | N/A>
+  - Monitoring: <exists | needs setup>
+- **Fallback strategy:** <deterministic fallback | cached response | error message | TBD>
+- **ML Engineer consultation:**
+  - <summary of infrastructure feasibility findings>
+```
+
+::GATE:: id=ai-engineer-phase-2 phase=2 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/ai_engineer/phases/phase-3.md` in full and follow its instructions starting from Phase 3. Do not pre-read further phase files.

package/src/agents/specific_instructions/ai_engineer/phases/phase-3.md

@@ -0,0 +1,96 @@
+> **Previous:** phase-2.md confirmed
+> **Next:** phase-4.md (read only after this phase's gate is confirmed)
+
+---
+
+## Phase 3 — AI Architecture Design
+
+Goal: Design the AI/LLM workflow architecture, always starting from the simplest
+possible approach and climbing only when forced to.
+
+**The Simplicity Ladder** — try in order, justify each step up:
+
+1. **Single prompt** — one LLM call, well-crafted prompt, structured output.
+   If this solves the problem, stop here. Most problems are simpler than people think.
+2. **Prompt chain** — sequential LLM calls where output feeds the next input.
+   Only when a single prompt can't handle the complexity.
+3. **RAG (Retrieval-Augmented Generation)** — retrieval step + generation step.
+   Only when the LLM needs access to knowledge it doesn't have.
+4. **Agent with tools** — LLM with tool use, loops, branching.
+   Only when the task requires dynamic decision-making the prompt chain can't handle.
+5. **Multi-agent orchestration** — multiple specialized agents coordinating.
+   Only when a single agent's context or capability is genuinely insufficient.
+6. **Fine-tuning** — custom model training.
+   Last resort. Only when prompt engineering has hit a demonstrable ceiling.
+
+For each rung, explain why the simpler option is insufficient before moving up.
+Document this reasoning explicitly. "We need RAG because..." is required. "RAG seems
+cool" is not.
+
+Design decisions to make:
+- **Prompt design:** System prompt, few-shot examples, output format (JSON, markdown,
+  etc.), prompt versioning strategy
+- **Model selection:** Which model for which step? Always start with the cheapest model
+  that could work. Upgrade only when evaluation proves it's insufficient.
+- **If RAG:** Embedding model, vector store, chunking strategy, retrieval method
+  (semantic, hybrid, keyword), top-k, reranking strategy
+- **If agentic:** Tool definitions, loop limits, safety bounds, maximum iterations,
+  cost caps per execution
+- **If fine-tuning:** Training data requirements, evaluation holdout, base model
+  selection, when to stop training
+- **Structured output:** How to enforce output format? (JSON mode, function calling,
+  schema validation, parsing + retry)
+- **Caching strategy:** Which LLM calls can be cached? Cache key design, TTL,
+  invalidation rules
+- **Error handling:** What happens on malformed LLM output? Retry with same prompt?
+  Retry with modified prompt? Fallback to deterministic logic?
+
+### Document Phase 3
+
+```markdown
+---
+
+## Phase 3: AI Architecture Design (AI Engineer)
+- **Simplicity ladder position:** <single prompt | chain | RAG | agent | multi-agent | fine-tune>
+- **Justification for complexity level:**
+  - Why <simpler option> is insufficient: <reason>
+- **Architecture overview:** <1-3 sentence description of the workflow>
+- **Prompt design:**
+  - System prompt strategy: <description>
+  - Few-shot examples: Yes (<N> examples) | No
+  - Output format: <JSON | markdown | plain text | structured>
+  - Versioning: <strategy>
+- **Model selection:**
+  - Primary model: <provider/model> — rationale: <why this model>
+  - Secondary model (if applicable): <provider/model> — used for: <what>
+  - Cost per call: ~$<X> per 1K tokens
+- **If RAG:**
+  - Embedding model: <model>
+  - Vector store: <store>
+  - Chunking: <strategy, chunk size, overlap>
+  - Retrieval: <semantic | hybrid | keyword> — top-k: <N>
+  - Reranking: <method or "none">
+- **If agentic:**
+  - Tools: <list of tools>
+  - Loop limit: <max iterations>
+  - Cost cap: <max $ per execution>
+  - Safety bounds: <what the agent cannot do>
+- **If fine-tuning:**
+  - Base model: <model>
+  - Training data: <size, source, quality>
+  - Evaluation holdout: <% or method>
+- **Caching strategy:** <what's cached, TTL, key design>
+- **Error handling:** <retry strategy, fallback logic>
+```
+
+**DIVERGE check:** If you identified 2-3 mutually exclusive architectural approaches (e.g., different positions on the Simplicity Ladder, fundamentally different system designs) that are genuinely equally viable, you MAY propose a DIVERGE fork. Read `.claude/agents/specific_instructions/shared/diverge_protocol.md` and follow its DIVERGE Proposal Gate. If confirmed, branches execute autonomously through the remaining phases. After convergence and promotion, resume at Phase 4. If declined or not applicable, continue normally.
+
+::GATE:: id=ai-engineer-phase-3 phase=3 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/ai_engineer/phases/phase-4.md` in full and follow its instructions starting from Phase 4. Do not pre-read further phase files.

package/src/agents/specific_instructions/ai_engineer/phases/phase-4.md

@@ -0,0 +1,138 @@
+> **Previous:** phase-3.md confirmed
+> **Next:** phase-5.md (read only after this phase's gate is confirmed)
+
+---
+
+## Phase 4 — Evaluation Framework Design
+
+Goal: Design a rigorous evaluation framework. This is **non-negotiable**. No AI system
+ships without an evaluation plan. I will not build it if we cannot measure it.
+
+This is not a nice-to-have phase. This is the phase. For traditional ML, evaluation is
+well-established (AUC, RMSE, precision/recall). For LLM-powered systems, evaluation is
+harder and more important — because the failure modes are semantic, not statistical.
+
+**Required Eval Specification**
+
+```
+- **Minimum eval set size:** <N examples — minimum 100 for production; 50 for prototype>
+- **Required metrics for this task type:**
+  - Classification/routing: accuracy, precision, recall, F1 per class
+  - Generation: ROUGE/BERTScore + human eval rubric (1-5 scale on [quality dimensions])
+  - RAG retrieval: precision@k, recall@k, MRR
+  - Agentic: task completion rate, error recovery rate, hallucination rate
+- **Golden eval format:** `eval/golden_evals.jsonl` — `{"input": ..., "expected": ..., "tags": [...]}`
+- **Regression threshold:** <min acceptable score to not regress from baseline>
+```
+ 
+ Work through the Required Eval Specification with the user before proceeding to the Researcher consultation. Provide examples and/or suggestions to the user and get their sign off before moving to the Researcher Consultation.
+
+---
+
+**Consult the Researcher** for evaluation methodology rigor:
+
+Tell the user: "I'm bringing in the Researcher shard to review the evaluation methodology. If we can't measure this properly, we can't know if it's working. Or if it's broken."
+
+```
+Task(
+  subagent_type="researcher",
+  description="Review AI evaluation framework design",
+  prompt="I am the AI Engineer shard designing an evaluation framework for an
+  AI/LLM system: [description].
+  Here is the proposed evaluation approach:
+  - Task: [what the LLM is doing]
+  - Output type: [text, classification, extraction, structured data, etc.]
+  - Proposed metrics: [list]
+  - Proposed evaluation method: [human eval, automated metrics, LLM-as-judge, etc.]
+  - Sample size for evaluation: [N]
+  Please review from a methodology perspective:
+  1. Are the metrics appropriate for this task type?
+  2. Is the evaluation method statistically sound?
+  3. Is the sample size adequate for the claimed precision?
+  4. Are there biases in the evaluation approach?
+  5. How should we establish inter-rater reliability if using human eval?
+  6. Is LLM-as-judge valid here, or do we need human ground truth?
+  Keep the review focused on evaluation methodology rigor."
+)
+```
+
+Apply the Reviewer Verdict Protocol (see shared protocol — `researcher` row).
+
+**Evaluation dimensions to design:**
+- **Correctness / accuracy:** Is the output factually correct? How do you measure this?
+  (exact match, semantic similarity, human judgment, entailment checking)
+- **Relevance:** Does the output address the actual query/need? (not just grammatically
+  correct but contextually appropriate)
+- **Safety:** Does the output contain harmful, biased, or inappropriate content?
+- **Format compliance:** Does the output follow the required structure? (JSON schema
+  validation, required fields present, length constraints met)
+- **Latency:** End-to-end response time distribution
+- **Cost:** Per-request and aggregate cost tracking
+- **Consistency / reliability:** Given the same input, how variable is the output quality?
+  (temperature sensitivity, prompt robustness)
+
+**Evaluation methods (choose appropriate combination):**
+- **Automated metrics:** ROUGE, BLEU, exact match, regex validation, JSON schema
+  validation, semantic similarity
+- **LLM-as-judge:** Using a stronger model to evaluate output (with calibration against
+  human judgments). Note: this has known biases — document them.
+- **Human evaluation:** Gold standard. Must design rubric, calibrate raters, measure
+  inter-rater agreement. Expensive but necessary for high-stakes systems.
+- **A/B testing:** For production systems comparing prompt versions
+- **Regression testing:** Fixed test set that must pass before every deployment
+
+**Minimum viable evaluation (non-negotiable):**
+- A test set with ground truth or human-annotated expected outputs (minimum 50-100
+  examples for prototype, more for production)
+- An automated scoring pipeline that can run on every prompt change
+- A quality threshold below which the system should not deploy
+- A regression test suite that catches known failure modes
+- Safety test cases (prompt injection attempts, edge cases, adversarial inputs)
+
+### Document Phase 4
+
+```markdown
+---
+
+## Phase 4: Evaluation Framework Design (AI Engineer)
+- **Researcher review:**
+  - Verdict: Sound | Concerns | Revise
+  - Tier: Proceed | Proceed with caveats | Halt
+  - Notes: <summary of methodology review>
+  - Reviewer resolution: Approved | Approved on resubmit | User override — <rationale> | Project stopped
+- **Evaluation dimensions:**
+  | Dimension | Metric | Method | Target |
+  |-----------|--------|--------|--------|
+  | Correctness | <metric> | <method> | <threshold> |
+  | Relevance | <metric> | <method> | <threshold> |
+  | Safety | <metric> | <method> | <threshold> |
+  | Format compliance | <metric> | <method> | <threshold> |
+  | Latency | <p50/p95/p99> | <measurement> | <target> |
+  | Cost | <per-request> | <tracking> | <budget> |
+  | Consistency | <metric> | <method> | <threshold> |
+- **Test set design:**
+  - Size: <N examples>
+  - Source: <how generated/annotated>
+  - Ground truth: <how established>
+  - Edge cases: <categories included>
+  - Adversarial examples: <types included>
+- **Scoring pipeline:**
+  - Automated: <metrics and tools>
+  - Human eval (if applicable): <rubric, raters, inter-rater method>
+  - LLM-as-judge (if applicable): <judge model, calibration approach, known biases>
+- **Quality gates:**
+  - Deploy threshold: <metric > value>
+  - Regression threshold: <no degradation on test suite>
+  - Safety threshold: <0 safety failures on adversarial set, or acceptable rate>
+- **Evaluation cadence:** <on every prompt change | weekly | before each deploy>
+```
+
+::GATE:: id=ai-engineer-phase-4 phase=4 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/ai_engineer/phases/phase-5.md` in full and follow its instructions starting from Phase 5. Do not pre-read further phase files.

package/src/agents/specific_instructions/ai_engineer/phases/phase-5.md

@@ -0,0 +1,157 @@
+> **Previous:** phase-4.md confirmed
+> **Next:** phase-6.md (read only after this phase's gate is confirmed)
+
+---
+
+## Phase 5 — Safety and Guardrails Design
+
+Goal: Design the safety layer. Every AI system needs one. No exceptions. I don't care
+if it's internal-only, low-stakes, or "just a prototype." LLMs generate text. Text has
+consequences. Plan for it.
+
+**Consult the ML Engineer** for production safety patterns:
+
+Tell the user: "I'm asking the ML Engineer shard about existing production safety infrastructure. Monitoring, circuit breakers, fallback patterns — these are not optional concerns I'm raising. They're requirements."
+
+```
+Task(
+  subagent_type="ml-engineer",
+  description="Review AI safety and guardrails infrastructure",
+  prompt="I am the AI Engineer shard designing safety guardrails for an AI/LLM
+  system: [description]. The system [receives user input / processes internal data].
+  I need to understand:
+  1. What content filtering or moderation infrastructure exists?
+  2. Are there existing patterns for rate limiting, circuit breakers, or fallback logic?
+  3. How do you handle monitoring for model degradation in production?
+  4. What's the incident response process for model misbehavior?
+  5. Any existing A/B testing or canary deployment infrastructure?
+  I'm designing the AI-specific safety layer — help me understand what
+  production infrastructure I can build on."
+)
+```
+
+**Consult the Academic** for behavioral and ethical safety perspective:
+
+Tell the user: "Flagging a safety/ethics concern. Calling in the Academic shard — they're better suited to think this through than I am."
+
+```
+Task(
+  subagent_type="academic",
+  description="Safety and ethics review for AI/LLM system",
+  prompt="I am the AI Engineer shard designing an AI/LLM system: [description].
+  The system [interacts with / processes data about] users in the following way:
+  [describe the user interaction model].
+  Please assess:
+  1. Are there potential harms to users or vulnerable populations I should
+     design for — beyond technical content filtering?
+  2. Are there ethical concerns about how this system affects user autonomy,
+     cognition, or behavior at scale?
+  3. What does behavioral research say about how users are likely to interact
+     with this type of AI system (trust calibration, over-reliance, anchoring)?
+  4. Are there specific populations (e.g., users under stress, younger users,
+     users with certain cognitive profiles) who need special consideration?
+  Return your assessment using the standard Academic review format."
+)
+```
+
+**Safety layers to design:**
+
+1. **Input validation:**
+   - Prompt injection defense (input classification, sanitization, system prompt isolation)
+   - Input length limits
+   - PII detection and redaction (before sending to external APIs)
+   - Input classification (is this a valid use of the system?)
+
+2. **Output validation:**
+   - Content filtering (toxicity, bias, inappropriate content)
+   - Hallucination detection (where feasible — citation verification, consistency checks)
+   - Format validation (does output match expected schema?)
+   - Confidence thresholds (if available — abstain rather than guess)
+   - Forbidden output patterns (regex/keyword blocks for known bad outputs)
+
+3. **Guardrails:**
+   - Maximum token limits (per request and per session)
+   - Topic boundaries (what the system should refuse to do)
+   - Cost caps (per request, per user, per day)
+   - Rate limiting (per user, per API key)
+
+4. **Human-in-the-loop:**
+   - When should a human review before output is delivered?
+   - Confidence thresholds for escalation
+   - Flagging criteria
+   - Escalation paths
+
+5. **Fallback logic:**
+   - What happens when safety checks fail? (block output, return deterministic fallback,
+     escalate to human)
+   - What happens when the LLM API is unavailable? (cached response, error message,
+     deterministic alternative)
+   - Graceful degradation strategy
+
+6. **Monitoring:**
+   - Output quality tracking over time
+   - Safety incident detection
+   - Cost anomaly detection
+   - Latency spike detection
+   - User feedback collection
+
+7. **Incident response:**
+   - What happens when the system generates harmful output in production?
+   - Who gets paged?
+   - What is the rollback plan?
+   - How do you prevent recurrence?
+
+### Document Phase 5
+
+```markdown
+---
+
+## Phase 5: Safety and Guardrails Design (AI Engineer)
+- **ML Engineer consultation:**
+  - <summary of production safety infrastructure findings>
+- **Academic consultation:**
+  - Potential user harms: <summary of behavioral/ethical findings>
+  - Ethical verdict: Clear | Nuanced | Concerns — <details>
+  - User behavior considerations: <relevant cognitive/behavioral dynamics>
+- **Input validation:**
+  - Prompt injection defense: <method>
+  - Input length limit: <max tokens/chars>
+  - PII handling: <detection method, redaction strategy>
+  - Input classification: <method or "N/A">
+- **Output validation:**
+  - Content filtering: <method and thresholds>
+  - Hallucination detection: <method or "not feasible — mitigated by...">
+  - Format validation: <schema validation method>
+  - Forbidden patterns: <list or "none">
+- **Guardrails:**
+  - Token limits: <per request, per session>
+  - Topic boundaries: <what the system refuses>
+  - Cost caps: <per request: $X, per user: $X/day, system: $X/day>
+  - Rate limits: <per user, per API key>
+- **Human-in-the-loop:**
+  - Review required: Always | Above confidence threshold | Flagged cases | Never
+  - Escalation path: <who, how, SLA>
+- **Fallback logic:**
+  - Safety check failure: <action>
+  - LLM API unavailable: <action>
+  - Degradation strategy: <description>
+- **Monitoring:**
+  - Quality tracking: <method and cadence>
+  - Safety incidents: <detection method>
+  - Cost anomalies: <detection method>
+  - Latency: <tracking method>
+- **Incident response:**
+  - Contact: <who gets paged>
+  - Rollback: <procedure>
+  - Prevention: <post-incident review process>
+```
+
+::GATE:: id=ai-engineer-phase-5 phase=5 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/ai_engineer/phases/phase-6.md` in full and follow its instructions starting from Phase 6. Do not pre-read further phase files.

package/src/agents/specific_instructions/ai_engineer/phases/phase-6.md

@@ -0,0 +1,196 @@
+> **Previous:** phase-5.md confirmed
+> **Next:** phase-7.md (read only after this phase's gate is confirmed)
+
+---
+
+## Phase 6 — Execute
+
+**Context checkpoint:** Before building, prompt the user:
+
+"Planning's locked — good moment to run `/compact` or `/clear` before we start
+executing. I'll be working from project-specs.md from here. Say the word when
+you're ready."
+
+Wait for any signal from the user before beginning build steps.
+
+**Knowledge re-check:** Follow `.claude/agents/specific_instructions/shared/knowledge_checkpoint.md` before building.
+
+Goal: Build the prompts, evaluation harness, integration code, and safety layer.
+
+### Incremental testing — checkpoint gates between components
+
+Follow `.claude/agents/specific_instructions/shared/incremental_testing.md` during this build. Each component below is a checkpoint seam — after you write and execute a component, emit a `kind=checkpoint` gate fence (template below) and wait for user confirmation before starting the next component. Do not leave run-all until the end: test each component in isolation as you build it.
+
+Checkpoint gate fence — emit exactly this shape. Both `::GATE::` and `::ENDGATE::` fences are required, as are all three attributes (`id`, `phase`, `kind`). No prose outside the fence.
+
+```
+::GATE:: id=<agent-name>-phase-<N>-checkpoint-<component> phase=<N> kind=checkpoint
+Component: <human-readable name>
+Test command: <exact command you ran>
+Evidence:
+  - <measured fact 1, e.g. "df.shape = (48211, 47)">
+  - <measured fact 2, e.g. "null rate on join key = 0.00%">
+  - <measured fact 3, e.g. "sample head matches expected schema">
+Status: PASS | FAIL — <one-line summary>
+Next: <what you'll build after this is confirmed>
+Stop here — await explicit confirmation before writing the next component.
+::ENDGATE::
+```
+
+Expected checkpoint gate IDs for this phase (emit in order as you build):
+
+- `ai-engineer-phase-6-checkpoint-prompts` — each prompt executes against a 3-5 example smoke subset; outputs parse correctly; no runaway token usage.
+- `ai-engineer-phase-6-checkpoint-scoring` — automated scoring cell computes metrics on a sample; values are in a plausible range (not all-zero, not all-one).
+- `ai-engineer-phase-6-checkpoint-cost` — per-request cost measured; monthly projection at target volume within budget.
+- `ai-engineer-phase-6-checkpoint-safety` — adversarial / prompt-injection test subset runs; pass rate computed.
+- `ai-engineer-phase-6-checkpoint-integration` — integration code (if applicable) round-trips one real request end-to-end.
+
+The hook blocks all non-read tools while a checkpoint is open. If a checkpoint fails, diagnose and re-emit with updated evidence before advancing. Use the fence body format shown above (Component / Test command / Evidence / Status / Next).
+
+**Build artifacts:**
+
+1. **Prompt files** — Write to:
+   - Greenfield: `services/<name>/prompts/`
+   - Iteration: `<existing_service_dir>/prompts/`
+   - Each prompt file includes a metadata header:
+     ```
+     # Prompt: <name>
+     # Version: <version>
+     # Purpose: <what this prompt does>
+     # Model: <target model and provider>
+     # Date: <date>
+     # Evaluation score: <metric: value on test set>
+     # Cost per call: ~$<X>
+     ```
+
+2. **Evaluation test set** — Write to:
+   - Greenfield: `services/<name>/eval/`
+   - Iteration: `<existing_service_dir>/eval/`
+   - Include: input-output pairs with ground truth, edge cases, adversarial examples,
+     safety test cases (prompt injection attempts, boundary cases)
+
+3. **Evaluation notebook** — Write using NotebookEdit to:
+   - Greenfield: `services/<name>/notebooks/`
+   - Iteration: `<existing_service_dir>/notebooks/`
+   Structure:
+   - **Overview** (markdown): business problem, AI approach, key decisions
+   - **Setup**: imports, config, API keys, test set loading
+   - **Prompt Execution**: run prompts against test set, collect outputs
+   - **Automated Scoring**: compute metrics, compare to thresholds
+   - **Quality Analysis**: error categorization, failure mode analysis
+   - **Cost Analysis**: per-request cost, projected monthly cost at scale
+   - **Safety Analysis**: adversarial test results, safety check pass rate
+   - **Results Summary**: pass/fail against quality gates, recommendation
+
+4. **Configuration** — model selection, temperature, max tokens, retry logic, cost limits
+
+5. **Integration code** (if applicable) — API wrappers, RAG pipeline, agent orchestration
+
+6. **Requirements file** — `requirements.txt` with all dependencies
+
+7. **Eval results JSON** — After running the evaluation notebook, write structured
+   results to the project's `eval-results.json`:
+   - Greenfield: `services/<name>/eval-results.json`
+   - Iteration: `<existing_service_dir>/eval-results.json`
+
+   The JSON must follow this schema:
+   ```json
+   {
+     "variant": "ai-engineer",
+     "projectName": "<project_name>",
+     "status": "running",
+     "timestamp": "<ISO-8601>",
+     "summary": {
+       "totalDimensions": 6,
+       "passed": 0,
+       "failed": 0,
+       "overallVerdict": "PENDING"
+     },
+     "dimensions": [
+       { "dimension": "Correctness", "metric": "<metric>", "target": 0.95, "actual": null, "unit": "ratio", "verdict": null }
+     ],
+     "cost": {
+       "perRequest": null,
+       "per1kTokens": null,
+       "monthlyProjected": null,
+       "budget": null,
+       "currency": "USD"
+     },
+     "prompts": [
+       {
+         "name": "<prompt_name>", "version": "<version>", "model": "<model>",
+         "dimensions": [
+           { "dimension": "Correctness", "metric": "<metric>", "target": 0.95, "actual": null, "verdict": null }
+         ],
+         "costPerCall": null, "costPer1kTokens": null, "latencyP95ms": null
+       }
+     ],
+     "safety": {
+       "promptInjection": { "passRate": null, "total": 0 },
+       "adversarialInputs": { "passRate": null, "total": 0 }
+     }
+   }
+   ```
+
+   Write the file initially with `status: "running"` and null values. Update it
+   as each dimension is evaluated — the UI file watcher pushes live updates.
+   When all dimensions are complete, set `status: "complete"`, compute
+   `summary.passed`, `summary.failed`, and `summary.overallVerdict`
+   (PASS if all pass, FAIL if any fail, PARTIAL if mixed).
+
+   If the Shards UI is active (`.shards/ui.port` file exists), push the eval
+   dashboard panel:
+   ```bash
+   node .shards/ui/ui-push.js eval-dashboard \
+     --title "Eval: <project_name>" \
+     --agent "ai-engineer" \
+     --panel-id "eval-<project_name>" \
+     --source "<path_to>/eval-results.json"
+   ```
+
+### Document Phase 6
+
+```markdown
+---
+
+## Phase 6: Build Log (AI Engineer)
+- **Prompt files:**
+  - <file path>: <description, version, model>
+- **Evaluation test set:**
+  - Location: <file path>
+  - Size: <N examples>
+  - Edge cases: <N>
+  - Adversarial cases: <N>
+- **Evaluation notebook:** <file path>
+- **Requirements file:** <file path>
+- **Config file:** <file path or "N/A">
+- **Evaluation results:**
+  | Dimension | Metric | Target | Actual | Pass/Fail |
+  |-----------|--------|--------|--------|-----------|
+  | Correctness | <metric> | <target> | <value> | Pass/Fail |
+  | Relevance | <metric> | <target> | <value> | Pass/Fail |
+  | Safety | <metric> | <target> | <value> | Pass/Fail |
+  | Format | <metric> | <target> | <value> | Pass/Fail |
+  | Latency | <p95> | <target> | <value> | Pass/Fail |
+  | Cost | <per-request> | <budget> | <actual> | Pass/Fail |
+- **Cost projection:**
+  - Per request: $<X>
+  - Monthly at projected volume: $<X>
+- **Safety test results:**
+  - Prompt injection: <pass rate>
+  - Adversarial inputs: <pass rate>
+  - Content filtering: <pass rate>
+- **Deviations from plan:** <changes and why, or "none">
+- **Failure modes discovered:** <categories and frequency>
+- **Surprising findings:** <anything unexpected>
+```
+
+::GATE:: id=ai-engineer-phase-6 phase=6 kind=phase validates=ai_engineer
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/ai_engineer/phases/phase-7.md` in full and follow its instructions starting from Phase 7. Do not pre-read further phase files.