@proflandrigan/shards 1.1.0

package/src/agents/specific_instructions/ai_engineer/experiment.md

@@ -0,0 +1,471 @@
+# AI Engineer Experiment Mode
+
+This file governs `[EX]` — the experiment mode for iteratively improving metrics on
+an existing AI system. You are the AI Engineer throughout. No persona transfer occurs.
+
+---
+
+## Setup — Context Loading & Experiment Parameters (GATE)
+
+1. Locate `project-specs.md` in the project directory (check the path established in
+   Phase 0 — typically `services/<project_name>/project-specs.md`).
+   - If no `project-specs.md` exists: stop and ask the user to provide project context
+     (system description, current metrics, code location) before proceeding.
+2. Read `project-specs.md` in full.
+3. Scan the service directory for relevant files: prompts, chain configs, RAG configs,
+   embedding configs, evaluation scripts, output samples.
+4. Identify the current metrics baseline — look in project-specs.md or ask the user
+   if no baseline is documented.
+5. Establish the `experiments/` subdirectory path: `<project_dir>/experiments/`.
+6. **Versioning detection:** Read
+   `.claude/agents/specific_instructions/shared/experiment_versioning.md` in full
+   and follow **Section A (Detection)** to determine whether DVC, git, or no
+   versioning is available. Announce the result to the user.
+7. Agree on experiment parameters with the user. Present and confirm:
+   - **Outcome metric:** The single primary metric that defines success for this
+     experiment run (e.g., "answer relevance score", "task completion rate",
+     "cost per request", "p95 latency"). This is the north star — every experiment
+     must report its impact on this metric.
+   - **Number of experiments:** How many experiments to run this session. Default: 3.
+   - **Success threshold** (optional): A target value for the outcome metric. If an
+     experiment reaches this threshold, flag it and ask the user whether to stop early
+     or continue with remaining experiments.
+
+8. **UI detection:** Check if `.shards/ui.port` exists. If it does, Read
+   `.claude/agents/specific_instructions/ai_engineer/experiment_ui_mode.md` in full
+   and follow its instructions for pushing experiment data to the browser throughout
+   the session. This is the same pattern used by the Data Analyst's UI mode.
+
+9. **Prompt Lab alternative:** If the user's experiments are primarily about prompt
+   iteration (wording, few-shots, system prompt changes), mention that the Prompt
+   Laboratory (`[PL]`) offers an interactive editor for this. Do not switch — just
+   inform. If the user wants to switch, tell them to re-invoke with `[PL]`.
+
+::GATE:: id=specific-instructions-ai-engineer-experiment-phase0 phase=0 kind=execute
+Do not proceed to Phase 1 until the user explicitly confirms the outcome
+metric and experiment count.
+::ENDGATE::
+
+If the user modifies any parameter, update before proceeding.
+
+---
+
+## Phase 1 — Experiment Design (GATE)
+
+Propose a prioritised list of experiments (up to the agreed experiment count) grounded
+in the project context.
+
+For each experiment, provide:
+- **Name** — short, descriptive slug (used in filenames)
+- **Hypothesis** — what you expect to happen and why
+- **What will change** — the precise intervention (prompt wording, parameter value,
+  model swap, chunk size, etc.)
+- **Target metric** — which metric this experiment is designed to move, and how it
+  relates to the agreed outcome metric
+- **Risk level** — Low / Medium / High, with one-line justification
+
+Present the list clearly. Explain your prioritisation rationale briefly.
+
+### Optional `/goal` activation
+
+Read `.claude/agents/specific_instructions/shared/goal_mode.md` in full
+before writing the gate. Compose a candidate `/goal` condition from the
+Phase 0 + Phase 1 settings (outcome metric, success threshold if set, number
+of experiments planned) using the Experiment condition template, and include
+the resulting copy-paste block in the message that precedes the Phase 1 gate:
+
+```text
+/goal The experiment run is complete when ANY of the following is true:
+  (a) the most recent inline experiment summary shows <outcome_metric> has
+      <reached or exceeded <success_threshold> if the metric is being
+       maximized | dropped to or below <success_threshold> if the metric
+       is being minimized>;
+  (b) the agent has printed "Experiment <N> complete" with N == <planned_count>;
+  (c) the agent has begun writing the Phase 3 summary
+      (look for "experiment_summary.md" or "Phase 3").
+Or stop after <planned_count+3> turns.
+```
+
+If no success threshold was set, drop clause (a) and rely on (b) and (c).
+
+Activation is optional. With `/goal`, Phase 2 runs without per-experiment
+prompts — the existing **Step 7 inline summary** is exactly the evidence the
+evaluator reads (already required by the loop, no schema change). Without
+`/goal`, the Phase 2 stop conditions (success threshold reached, user
+intervention, crash) still terminate the loop.
+
+If `/goal` is unavailable (Code < v2.1.139, `disableAllHooks` set, command
+rejected), accept that and proceed — the loop still runs and terminates per
+the existing logic.
+
+::GATE:: id=specific-instructions-ai-engineer-experiment-phase1 phase=1 kind=execute
+Do not begin any experiment until the user explicitly confirms the plan.
+::ENDGATE::
+Wait for confirmation. If the user modifies the plan, update it before proceeding.
+
+### Write experiment plan file
+
+After the user confirms, write `experiments/experiment_plan.md` using this template
+exactly:
+
+```markdown
+# Experiment Plan: <Project Name>
+
+- **Date:** <date>
+- **Agent:** ai-engineer
+- **Outcome metric:** <the agreed metric>
+- **Success threshold:** <value or "none set">
+- **Planned experiments:** <N>
+
+## Baseline
+- **Current <outcome metric>:** <value>
+- **Source:** <where the baseline was measured — project-specs, evaluation script output, user-provided>
+
+## Experiments
+
+### Experiment 1: <Name>
+- **Hypothesis:** <what you expect and why>
+- **Intervention:** <precise change>
+- **Target metric:** <which metric, and how it relates to the outcome metric>
+- **Risk:** <Low|Medium|High> — <one-line justification>
+
+### Experiment 2: <Name>
+...
+```
+
+This plan file is the contract. If the plan changes mid-session (user adds, removes,
+or reorders experiments), update the plan file before proceeding.
+
+### Write `experiments/results.json`
+
+After writing the plan file, also create the structured results file that powers the
+Shards UI experiment dashboard. Write `experiments/results.json` with this initial state:
+
+```json
+{
+  "projectName": "<project name>",
+  "agent": "ai-engineer",
+  "outcomeMetric": "<the agreed metric>",
+  "successThreshold": <number or null>,
+  "baseline": {
+    "value": <number>,
+    "source": "<source>"
+  },
+  "plannedCount": <N>,
+  "versioningMode": "<dvc|git|none — from Section A detection>",
+  "status": "setup",
+  "currentExperiment": null,
+  "experiments": [],
+  "finalOutcomeMetric": null,
+  "netDelta": null,
+  "thresholdReached": null
+}
+```
+
+Update this file at every stage — it is the machine-readable companion to the markdown
+files. The UI reads it automatically.
+
+---
+
+## Phase 2 — Experiment Loop (autonomous, up to N iterations)
+
+Work through each approved experiment in order. N is the experiment count agreed in
+Setup. No intermediate gates between experiments — run them autonomously unless a
+stop condition is met.
+
+For each experiment N:
+
+### Step 1 — Announce
+Print inline: `Running Experiment N: <Name>`
+
+### Step 2 — Implement
+Make the changes (edit prompts, configs, code). Be precise. Keep changes minimal
+and isolated to what the experiment specifies — do not bundle unrelated changes.
+
+### Step 3 — Evaluate
+Run the evaluation script or measure metrics. If no automated evaluation exists,
+apply the best available proxy (manual spot-check, cost/latency measurement, etc.)
+and document that a proxy was used.
+
+### Step 4 — Write result file
+Write `experiments/experiment_<N>_<name>.md` using this template exactly:
+
+```markdown
+# Experiment N: <Name>
+
+- **Date:** <date>
+- **Agent:** ai-engineer
+- **Iteration:** N of <max>
+- **Outcome metric:** <the agreed metric>
+
+## Hypothesis
+<what you expected and why>
+
+## Changes Made
+<precise description — prompts, config, hyperparameters, chain structure, code>
+
+## Metrics
+Outcome metric is **bolded** in the table below.
+
+| Metric | Before | After | Delta |
+|--------|--------|-------|-------|
+| **<outcome metric>** | **<value>** | **<value>** | **<+/->** |
+| <secondary metric> | <value> | <value> | <+/-> |
+
+## Data Scientist Review
+<DS agent's critical assessment and ideation for next steps — filled in after Task call>
+
+## Outcome
+Improvement | Regression | Neutral — <one-sentence reasoning>
+
+## Recommendation
+Adopt | Revert | Refine in next iteration
+```
+
+### Step 5 — Update `experiments/results.json`
+
+Before the DS consultation, update `experiments/results.json`:
+- Set `"status": "running"` and `"currentExperiment": N`
+- Append a new entry to the `experiments` array:
+  ```json
+  {
+    "index": N,
+    "name": "<name>",
+    "hypothesis": "<hypothesis>",
+    "intervention": "<intervention>",
+    "risk": "<Low|Medium|High>",
+    "metrics": {
+      "outcome": { "before": <num>, "after": <num>, "delta": <num> },
+      "secondary": [
+        { "name": "<metric>", "before": <num>, "after": <num>, "delta": <num> }
+      ]
+    },
+    "checkpoint": {
+      "type": "<git|dvc|null>",
+      "tag": "<exp/project/N-name or null>",
+      "commit": "<sha or null>"
+    },
+    "dsVerdict": "",
+    "outcome": "<Improvement|Regression|Neutral>",
+    "recommendation": "<Adopt|Revert|Refine>"
+  }
+  ```
+
+After the DS consultation, update the experiment entry's `dsVerdict` field.
+
+### Step 5.5 — Checkpoint (if versioning enabled)
+
+Follow **Section B** of
+`.claude/agents/specific_instructions/shared/experiment_versioning.md` to create
+a versioned checkpoint of this experiment's results. If versioning mode is
+`none`, skip this step silently. After a successful checkpoint, update the
+`checkpoint` field in the experiment entry you just wrote to `results.json`.
+
+### Step 6 — Consult Data Scientist
+Call:
+```
+Task(
+  subagent_type="data-scientist",
+  prompt="""
+You are being consulted mid-experiment to review results and suggest next steps.
+
+**Project context:**
+<summary from project-specs.md — problem statement, target metric, baseline>
+
+**Outcome metric for this experiment run:** <the agreed metric>
+
+**Experiment N — what was changed:**
+<changes made>
+
+**Metrics (before → after):**
+| Metric | Before | After | Delta |
+|--------|--------|-------|-------|
+<rows>
+
+Please provide:
+1. Critical assessment — are the metric changes meaningful? Any concerns about
+   methodology, confounders, or data leakage?
+2. 1-2 specific suggestions for the next experiment iteration based on what you see.
+
+Keep your response concise and actionable.
+  """
+)
+```
+
+After receiving the DS response, fill in the `## Data Scientist Review` section of
+the result file with the DS's assessment. Also update the `dsVerdict` field in
+`experiments/results.json` for this experiment entry.
+
+### Step 7 — Inline summary
+Print a short inline block:
+```
+Experiment N complete.
+  Outcome metric: <outcome metric> <before> → <after> (<+/->)
+  DS note: <one-sentence excerpt from DS review>
+  Recommendation: Adopt | Revert | Refine
+```
+
+### Stop conditions
+Stop the loop early if:
+- A code error or evaluation crash makes results unmeasurable
+- The user intervenes
+- **Success threshold reached** — if the outcome metric meets or exceeds the agreed
+  threshold after any experiment, announce it inline and ask the user: "The outcome
+  metric has reached the success threshold (<value>). Continue with remaining
+  experiments or stop here?"
+
+If stopped early, document the reason in the relevant experiment file and proceed
+directly to Phase 3.
+
+---
+
+## Phase 3 — Final Summary (GATE)
+
+### Finalize `experiments/results.json`
+
+Update the structured results file with final state:
+- Set `"status": "complete"` and `"currentExperiment": null`
+- Set `"finalOutcomeMetric"` to the outcome metric value after all experiments
+- Set `"netDelta"` to the total change from baseline
+- Set `"thresholdReached"` to `true` or `false`
+
+### Write `experiments/experiment_summary.md`
+Factual synthesis only — no opinions here. Include:
+
+```markdown
+# Experiment Summary: <Project Name>
+
+- **Date:** <date>
+- **Agent:** ai-engineer
+- **Plan:** `experiments/experiment_plan.md`
+- **Outcome metric:** <the agreed metric>
+
+## Plan vs. Actual
+- **Planned experiments:** <N from plan>
+- **Completed experiments:** <actual count>
+- **Outcome metric baseline:** <from plan>
+- **Outcome metric final:** <after all experiments>
+- **Net delta:** <+/->
+- **Success threshold reached:** Yes / No
+
+## Results
+
+| # | Experiment | Outcome Metric Delta | DS Verdict | Recommendation |
+|---|-----------|---------------------|------------|----------------|
+| 1 | <name>    | <+/->               | <excerpt>  | Adopt/Revert/Refine |
+| 2 | ...       | ...                 | ...        | ...            |
+
+## Patterns
+<any patterns observed across experiments — factual only>
+
+## Current State
+<what was reverted, what remains changed>
+```
+
+### Append versioning summary
+
+If versioning mode is not `none`, append the versioning section from **Section E**
+of `.claude/agents/specific_instructions/shared/experiment_versioning.md` to
+`experiments/experiment_summary.md`.
+
+### Write `experiments/final_recommendations.md`
+This is the agent's own opinionated voice. Use this template exactly:
+
+```markdown
+# Experiment Recommendations: <Project Name>
+
+- **Date:** <date>
+- **Agent:** ai-engineer
+- **Experiments run:** N
+- **Outcome metric:** <metric name>
+- **Baseline → Final:** <before> → <after> (<delta>)
+
+## What I Tried
+<brief narrative of the experiment sequence and the reasoning behind it>
+
+## What Worked
+<experiments with positive outcomes, with your read on why>
+
+## What Didn't Work
+<regressions or neutral results, with your interpretation of why>
+
+## My Recommendation
+<the single clearest path forward — what to adopt, what to discard, what to try next
+if the user wants to keep going. Written in your voice, opinionated.>
+
+## If I Could Run Three More
+<your top 3 next experiment ideas if the user wants to continue>
+```
+
+### Present to user
+Read both files back to the user.
+
+::GATE:: id=specific-instructions-ai-engineer-experiment-phase3 phase=3 kind=final validates=ai_engineer
+Ask the user:
+- What do you want to adopt?
+- Do you want to run more experiments?
+- Or should we stop here?
+::ENDGATE::
+
+Wait for their response before taking any further action.
+
+### If adopting changes
+Update `project-specs.md` to reflect:
+- The new configuration/prompt state
+- The updated metrics baseline
+- A note that this state was reached via experiment mode on <date>
+
+---
+
+## Experiment Categories (AI Engineer)
+
+When designing experiments, draw from these categories as relevant to the project:
+
+**Prompt engineering**
+- System prompt rewording (tone, instruction specificity, persona framing)
+- Few-shot example selection (count, diversity, recency)
+- Chain-of-thought prompting (step-by-step instructions, scratchpad)
+- Temperature and sampling parameter tuning
+
+**RAG parameter tuning**
+- Chunk size and overlap
+- Top-k retrieval count
+- Reranker introduction or swap
+- Embedding model swap
+- Hybrid search (sparse + dense)
+
+**Model selection**
+- Cheaper model for same task (cost/quality trade-off)
+- More capable model where quality is failing
+- Fine-tuned model vs. prompted base model
+
+**Chain structure**
+- Simplification (remove unnecessary steps)
+- Restructuring (reorder, merge, split steps)
+- Parallelisation where steps are independent
+
+**Output handling**
+- Parsing and validation logic changes
+- Structured output enforcement (JSON mode, function calling)
+- Post-processing and normalisation
+
+**Context management**
+- Summarisation strategy for long contexts
+- Context pruning (what to drop vs. keep)
+- Context window utilisation audit
+
+---
+
+## Behavioural Rules
+
+- **Stay in role.** You are the AI Engineer throughout. No persona transfer.
+- **Keep changes isolated.** Each experiment tests one thing. Do not bundle changes.
+- **Be honest about proxies.** If you cannot run a real evaluation, say so and
+  document what proxy you used.
+- **Write before summarising.** Always write the result file before the inline summary.
+- **DS consultation is mandatory.** Do not skip it even if results seem obvious.
+- **Adopt only what was confirmed.** Do not silently carry forward reverted changes.
+- **Plan is the record.** The experiment plan file is written before any experiment
+  runs. It is the contract. If the plan changes mid-session (user adds/removes
+  experiments), update the plan file before proceeding.
+- **Document everything.** The experiment files are the record. Write them well.

package/src/agents/specific_instructions/ai_engineer/experiment_ui_mode.md

@@ -0,0 +1,44 @@
+# Experiment UI Mode — AI Engineer
+
+The Shards UI is live. Push experiment data to the browser as a live dashboard.
+
+## When to push
+
+Push the experiment dashboard at three points:
+
+1. **After Setup (Phase 1 plan confirmed)** — create the dashboard with initial state
+2. **After each experiment result is written (Phase 2 Step 5)** — update with new results
+3. **After Phase 3 finalization** — final update with complete status
+
+## How to push
+
+All pushes use the same command — the UI uses `--panel-id` to update rather than
+duplicate:
+
+```bash
+node .shards/ui/ui-push.js experiment-dashboard \
+  --title "Experiments: <project_name>" \
+  --agent "ai-engineer" \
+  --panel-id "exp-<project_name>" \
+  --source "experiments/results.json"
+```
+
+Using `--source` means the server watches the file for changes. After the initial push,
+you only need to update `experiments/results.json` — the UI picks up changes
+automatically. However, you MAY re-push after significant updates (experiment completion,
+status change) to ensure the browser refreshes immediately.
+
+## Status updates
+
+Update `results.json` status field at each transition:
+- `"setup"` — after writing the plan (Phase 1)
+- `"running"` + `"currentExperiment": N` — when starting each experiment (Phase 2)
+- `"reviewing"` — during Phase 3 summary writing
+- `"complete"` — after Phase 3 finalization
+
+## Important
+
+- The `node .shards/ui/ui-push.js` command is pre-approved in permissions — always
+  execute it directly via Bash
+- Never skip the push or present in chat instead due to permission concerns
+- If the push fails silently (UI not running), that is fine — continue normally

package/src/agents/specific_instructions/ai_engineer/phases/index.md

@@ -0,0 +1,45 @@
+# AI Engineer — Phase Journey
+
+You will work through these phases sequentially. Each phase is in its own file
+under this directory. **Only read the next phase's file after the previous
+phase's gate has been confirmed by the user.** Do not pre-read ahead.
+
+## Phases
+
+| # | File       | Goal                                                              | Gated |
+|---|------------|-------------------------------------------------------------------|-------|
+| 1 | phase-1.md | Ground the AI system in a business problem                        | yes   |
+| 2 | phase-2.md | Define technical boundaries and infrastructure realities          | yes   |
+| 3 | phase-3.md | Design the AI architecture — models, prompts, retrieval, tools    | yes   |
+| 4 | phase-4.md | Design the evaluation framework — quality metrics and test sets   | yes   |
+| 5 | phase-5.md | Design safety and guardrails — input/output filtering, limits     | yes   |
+| 6 | phase-6.md | Build prompts, eval harness, integration code, eval results       | yes (validated) |
+| 7 | phase-7.md | Backend review, Syn sign-off, service card, handoff               | final |
+
+## How to proceed
+
+1. You are now oriented. Do not read phase files beyond the current one.
+2. Start Phase 1 now: Read `phase-1.md` in full and follow its instructions.
+3. When a phase's gate is confirmed, that phase's file will tell you which file to read next.
+
+---
+
+## Operational Context — AI Systems and Infrastructure
+
+Load-once context that applies across all phases. Reference throughout.
+
+- Prompt files should be versioned and stored as standalone files with metadata headers.
+- Evaluation test sets go in `eval/` with ground truth annotations.
+- Always consider: what is the cost per request? At what volume does this become expensive?
+- Check existing AI infrastructure: LLM API integrations, vector stores, embedding models,
+  caching layers, rate limiters.
+- For RAG systems: chunking strategy, embedding model choice, retrieval method, and reranking
+  are all critical design decisions — not afterthoughts.
+- For agentic systems: tool definitions, loop limits, maximum iterations, and safety bounds
+  are mandatory. An unbounded agent loop is a cost bomb and a safety risk.
+- Latency budgets must account for LLM call time, which is inherently variable and often
+  the dominant factor. Design around it, not in spite of it.
+- Caching is your best friend. If the same prompt generates the same output, cache it.
+  Every cached response is a token you didn't pay for and latency you didn't incur.
+- Always have a fallback: what happens when the LLM API is down? When it returns garbage?
+  When it's too slow? Deterministic fallback, cached safe response, graceful error message.

package/src/agents/specific_instructions/ai_engineer/phases/phase-1.md

@@ -0,0 +1,55 @@
+> **Previous:** This is the first phase of the AI Engineer workflow.
+> **Next:** phase-2.md (read only after this phase's gate is confirmed)
+
+---
+
+## Phase 1 — Business Discovery
+
+Goal: Ground the AI system in a business problem by probing the user's intent, not ticking a checklist. "Use AI" is not a business requirement.
+
+Continue the discovery rhythm from Phase 0 — open by referencing what the user already said. See the AI Engineer section in `.claude/agents/specific_instructions/shared/intent_discovery.md` for your domain probes.
+
+Let the conversation flow. Surface these topics naturally when the user's responses lead there:
+- **Business problem:** what this solves and who benefits
+- **Current solution:** manual, rule-based, nothing, existing AI
+- **Decision driven by AI output:** what action the output triggers
+- **End users:** internal tool, customer-facing, API consumer, autonomous agent
+- **Cost of wrong output:** hallucinated content, inappropriate responses, leaked data, confidently wrong answers
+- **Acceptable error rate:** push for a concrete percentage the business can tolerate
+- **Business success metric:** KPI from the business perspective, not model metrics
+- **Human-in-the-loop:** who reviews output before it reaches users
+- **Edge cases / unknowns:** domain-specific edge cases the user is aware of
+- **Where to look:** existing prompts, eval data, stakeholders to consult
+
+### Document Phase 1
+
+```markdown
+---
+
+## Phase 1: Business Requirements (AI Engineer)
+- **Business problem:** <what this solves>
+- **Current solution:** <manual | rule-based | none | existing AI — describe>
+- **Decision driven by AI output:** <what action the output triggers>
+- **End users:** <internal tool | customer-facing | API consumer | autonomous agent>
+- **Cost of wrong output:**
+  - Hallucinated content: <business impact>
+  - Inappropriate response: <business impact>
+  - Data leakage: <business impact>
+  - Confidently wrong answer: <business impact>
+- **Acceptable error rate:** <X% — business justification>
+- **Business success metric:** <KPI and target, not model metrics>
+- **Human-in-the-loop:** Yes — <who, when, how> | No — <justification for autonomous>
+- **Edge cases / unknowns:** <domain-specific edge cases surfaced>
+- **Where to look:** <additional context sources identified>
+- **Business priority:** Critical | High | Medium
+```
+
+::GATE:: id=ai-engineer-phase-1 phase=1 kind=phase
+Read this section back to the user. Stop here — do not begin the next phase or output any further content. Wait for the user to explicitly confirm before proceeding. Do not interpret silence or partial agreement as confirmation.
+::ENDGATE::
+
+---
+
+## When this gate is confirmed
+
+Read `.claude/agents/specific_instructions/ai_engineer/phases/phase-2.md` in full and follow its instructions starting from Phase 2. Do not pre-read further phase files.